From patchwork Mon Dec 11 15:56:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487435 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89CC3C10F05 for ; Mon, 11 Dec 2023 15:57:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DF3A16B00FA; Mon, 11 Dec 2023 10:57:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DA5FD6B00FC; Mon, 11 Dec 2023 10:57:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B0C146B00FD; Mon, 11 Dec 2023 10:57:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A004C6B00FA for ; Mon, 11 Dec 2023 10:57:04 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 77F241C1009 for ; Mon, 11 Dec 2023 15:57:04 +0000 (UTC) X-FDA: 81554991168.05.0CF5F45 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf24.hostedemail.com (Postfix) with ESMTP id A303118001F for ; Mon, 11 Dec 2023 15:57:02 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=byyfSTke; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf24.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310222; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=W84BJOqs6odi3w10Q8+aDyiUulYqaQMW45jL1XIh9aE=; b=ADuqVOtLLejtmT1t/BMOFJaYCg8w/nCfcM2Y2Kya6Nz/8k3PAN7xTgWud1fcYnsd2ugFza /NS84iASP9JGcNVA9IrEwtmZgjc6pXLioQU6wODG2GrEP+WpaFUkwkJ2bmI+xLLvcyd9JO 0cxyiQGV/Boz+Hv5u0la9oHOUneU3wg= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=byyfSTke; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf24.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310222; a=rsa-sha256; cv=none; b=DvTBnjbfjmkwTQD84Fu3hPdtY2rg3Y+cxdGywNtztSDxeYE8Kfnk5YSd//pLTblswB52z8 VPz79P4dLEV+iiMj/Yx+HlSmFg/7aDWKO13I5zbt8UqohcTS8m7xoYAvQMPbq5RkdJevTU /pgRJtREEE9xyUOIHYAk1aBah/47iG4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310221; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=W84BJOqs6odi3w10Q8+aDyiUulYqaQMW45jL1XIh9aE=; b=byyfSTkeoxVdqg+p1M4txTYuTU8w5mr21cUHYleyAdtdG51EdoyV65gPZQcbvSra1DXICy KMlDYFYZR0E+L/KcXIAIFm091fnfpFf1RjiTfIT/2p14FSY3qODlZczSnoGr1i8Pg/nhwf xEnhKMxjjc3YkmgeFkbsgiXqBUPV0Ck= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-518-De3rUVWSN9y0uphkMhN_QQ-1; Mon, 11 Dec 2023 10:56:59 -0500 X-MC-Unique: De3rUVWSN9y0uphkMhN_QQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9BC94870829; Mon, 11 Dec 2023 15:56:58 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 457E61121306; Mon, 11 Dec 2023 15:56:56 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu , Muchun Song Subject: [PATCH v1 01/39] mm/rmap: rename hugepage_add* to hugetlb_add* Date: Mon, 11 Dec 2023 16:56:14 +0100 Message-ID: <20231211155652.131054-2-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: A303118001F X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: yokycz5mhxfefpmq1jpxudw6rhseb9mg X-HE-Tag: 1702310222-502271 X-HE-Meta: U2FsdGVkX1/xTLWLhUjD0DlCS4QfUhK8rYpEL9BmjN3OTGzeGlAOiLlQmrMGqsntQpWY2RxtjqGOKPoOUSwO06Vqebk2R9hMIlH7N0/yDkq6CuVbZNE2BDHmKnxIH4Z6yZJkx7H6Hm3zcE5n1IZQv92L2f+7rVkiiPwBzXsoye5ZztJXwvZqCpi/VAlY5+zJ1wyZzDEoZRiMCTXZ3UIO2VyA5kUPX3Ip8Q88QlxfIjucAx8U+WI1NudUz1c9Y0dDtPYJnIRLBAGgKwf/XzN+xp4fBdnpjkPiWX61xllHedPhnLJvLasHPM3eDZ6Smv9hexIq4K6C15MIl23QgP2JkePQTPC68BdESqQ8LlS51xNH7CbWkVM5dR3M5Uq/EjSODGZznWJuDFOBrCaBZi1N7757tlRfLxRznKd0gjx/vDuF2l7HH/DUAbceqg94fRVHWVdfvfkbXQVF59vVezKZFAwoPD9ZWLcVhWb9RDAkgye8ljEIPZnnCXWnfdFooOpvqbnFfF6L5MUe5alnYLEwfK1Y36qt0QIbyft+fOzADtgstKa6acJZ0/Z3BEds8JbhY2l47ouSx00ihquyvCSI+TU/L0SAL3votk21KhXEB5EDMq3wakUv1CsHUNopoMqkUt6bD55w0VJJ6aY7GD1l+5Z+aW28I012TjO2l+FUMFkOfpNjixFEgNTj1DXG2OfWO5Px3c3PpxKfpaelconeHtS9vjd+QTED7QAf2kz+ZUfC58/NT26xidxx9c/vI8eY2/f0ISezp+5/aCf7gs+MHkukKAP68rc36BkZqr6TbuzCfNy21M6izjsXl0gbqSntY59tuTVeW3/GW5sQUeudxq6/5gVGE7jWNqLgNN2YzO6pUbxvC5J8xBfw34ItIdjBcZW2g7I1yWrBbFmxn0+wLVDjrDQ5jZPmeXzWBo3r6g1dnRsv6uf5mI4xqO7/uwkSis8UG9fBGGYEqidbwsf Q8Dfq45j OUZIwLi+FhGeR3/0vnr6Dtf5cGccjCmIJyPgFN4v+68aPMJfy3s0GkH7uT3koq7vuhcslhuWjFxJzo+3g8WjkXShpZkjVr4i2chAfL0CypLzJM/NHJuIKNowd8SXC7OImA3zizFpyxyFTSLqnn7AhqxhYyZ7AbRnQj1OTKC/C1ZRr6JY38nZyDCEcBzTkrTNp5kddYvJLQcuyZHdSZB1lp+JQbhpDIJLTCfpDJjvrCkIl6OBwUN+09uh+WmKdqgszWxXnZTDz9ACa631ap3OO2j5qLw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's just call it "hugetlb_". Yes, it's all already inconsistent and confusing because we have a lot of "hugepage_" functions for legacy reasons. But "hugetlb" cannot possibly be confused with transparent huge pages, and it matches "hugetlb.c" and "folio_test_hugetlb()". So let's minimize confusion in rmap code. Reviewed-by: Muchun Song Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts Reviewed-by: Matthew Wilcox (Oracle) --- include/linux/rmap.h | 4 ++-- mm/hugetlb.c | 8 ++++---- mm/migrate.c | 4 ++-- mm/rmap.c | 8 ++++---- 4 files changed, 12 insertions(+), 12 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index af6a32b6f3e7..0bfea866f39b 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -208,9 +208,9 @@ void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr, void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); -void hugepage_add_anon_rmap(struct folio *, struct vm_area_struct *, +void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address, rmap_t flags); -void hugepage_add_new_anon_rmap(struct folio *, struct vm_area_struct *, +void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); static inline void __page_dup_rmap(struct page *page, bool compound) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 6feb3e0630d1..305f3ca1dee6 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5285,7 +5285,7 @@ hugetlb_install_folio(struct vm_area_struct *vma, pte_t *ptep, unsigned long add pte_t newpte = make_huge_pte(vma, &new_folio->page, 1); __folio_mark_uptodate(new_folio); - hugepage_add_new_anon_rmap(new_folio, vma, addr); + hugetlb_add_new_anon_rmap(new_folio, vma, addr); if (userfaultfd_wp(vma) && huge_pte_uffd_wp(old)) newpte = huge_pte_mkuffd_wp(newpte); set_huge_pte_at(vma->vm_mm, addr, ptep, newpte, sz); @@ -5988,7 +5988,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* Break COW or unshare */ huge_ptep_clear_flush(vma, haddr, ptep); page_remove_rmap(&old_folio->page, vma, true); - hugepage_add_new_anon_rmap(new_folio, vma, haddr); + hugetlb_add_new_anon_rmap(new_folio, vma, haddr); if (huge_pte_uffd_wp(pte)) newpte = huge_pte_mkuffd_wp(newpte); set_huge_pte_at(mm, haddr, ptep, newpte, huge_page_size(h)); @@ -6277,7 +6277,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, goto backout; if (anon_rmap) - hugepage_add_new_anon_rmap(folio, vma, haddr); + hugetlb_add_new_anon_rmap(folio, vma, haddr); else page_dup_file_rmap(&folio->page, true); new_pte = make_huge_pte(vma, &folio->page, ((vma->vm_flags & VM_WRITE) @@ -6732,7 +6732,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, if (folio_in_pagecache) page_dup_file_rmap(&folio->page, true); else - hugepage_add_new_anon_rmap(folio, dst_vma, dst_addr); + hugetlb_add_new_anon_rmap(folio, dst_vma, dst_addr); /* * For either: (1) CONTINUE on a non-shared VMA, or (2) UFFDIO_COPY diff --git a/mm/migrate.c b/mm/migrate.c index 35a88334bb3c..4cb849fa0dd2 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -249,8 +249,8 @@ static bool remove_migration_pte(struct folio *folio, pte = arch_make_huge_pte(pte, shift, vma->vm_flags); if (folio_test_anon(folio)) - hugepage_add_anon_rmap(folio, vma, pvmw.address, - rmap_flags); + hugetlb_add_anon_rmap(folio, vma, pvmw.address, + rmap_flags); else page_dup_file_rmap(new, true); set_huge_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte, diff --git a/mm/rmap.c b/mm/rmap.c index 846fc79f3ca9..80d42c31281a 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2625,8 +2625,8 @@ void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc) * * RMAP_COMPOUND is ignored. */ -void hugepage_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, - unsigned long address, rmap_t flags) +void hugetlb_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, + unsigned long address, rmap_t flags) { VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); @@ -2637,8 +2637,8 @@ void hugepage_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, PageAnonExclusive(&folio->page), folio); } -void hugepage_add_new_anon_rmap(struct folio *folio, - struct vm_area_struct *vma, unsigned long address) +void hugetlb_add_new_anon_rmap(struct folio *folio, + struct vm_area_struct *vma, unsigned long address) { BUG_ON(address < vma->vm_start || address >= vma->vm_end); /* increment count (starts at -1) */ From patchwork Mon Dec 11 15:56:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487436 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6577CC4167B for ; Mon, 11 Dec 2023 15:57:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 038926B00FC; Mon, 11 Dec 2023 10:57:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F2D4C6B00FE; Mon, 11 Dec 2023 10:57:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D2EE16B00FF; Mon, 11 Dec 2023 10:57:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id BFAA06B00FC for ; Mon, 11 Dec 2023 10:57:07 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 5D70EA07BD for ; Mon, 11 Dec 2023 15:57:07 +0000 (UTC) X-FDA: 81554991294.04.DFBC6BB Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 965A140009 for ; Mon, 11 Dec 2023 15:57:05 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DSmm0QUn; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310225; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=K3NaOeaUC8VPVD4C/LE70HnwI+FiYJElEMKNgi9W/YQ=; b=Kpv+VH7upHDt8tmR4SIPshYFtAdWwEjD0LsICs8Hcf814glkdgFCsgJMPa4Dk75rodnSyE mFPFXq9BmsSMx1UW99306njVfSalULhgkUEwAB2LdWRXi6y/y2MjL+rlayEkYe78euWZZG VX++qKcgiI40BUgvyoE+MaSwt7ksXag= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DSmm0QUn; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310225; a=rsa-sha256; cv=none; b=cFB3QAgkeZK6sjQTNLuEVCtIug/Gg5m95g0+PHxOpDzsv7dqr6RrQi6bZxmoAvZbC3Knux ozDfTsDYk/AEC0B1myOLhW5ncbcqcqXDoEQ0o5H+V9IfclM6X5x+MM5EG8sORrOu56I7K8 X1On4OeQfgr6GcZFz9vvw8pxeegSYjU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310224; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=K3NaOeaUC8VPVD4C/LE70HnwI+FiYJElEMKNgi9W/YQ=; b=DSmm0QUn5tIDl1bONKY0NjzLlsAFkmNKuG7Da5DzbLDOnRTwN0/XIEhrcVKH14XldKrUTI DI4CY1bbQb7nNTAhF1xX8tCHQzCn5+sELdqHBVq9zTERaeTOf1I6+nvh1K6Ima2Ro6anp9 lAJPjE5oQfY6N1XIxQsjqLTyVBKPj6A= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-88-dxPz7bySN2KzjJFO1gbBAg-1; Mon, 11 Dec 2023 10:57:02 -0500 X-MC-Unique: dxPz7bySN2KzjJFO1gbBAg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BF1A0280A9BB; Mon, 11 Dec 2023 15:57:01 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id EA3491121306; Mon, 11 Dec 2023 15:56:58 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 02/39] mm/rmap: introduce and use hugetlb_remove_rmap() Date: Mon, 11 Dec 2023 16:56:15 +0100 Message-ID: <20231211155652.131054-3-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspam-User: X-Stat-Signature: iun99z89ndfb8ds5prw6i8soikay8sfu X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 965A140009 X-HE-Tag: 1702310225-9231 X-HE-Meta: U2FsdGVkX1+cRWeHSsP/6sT9o/cZHlGiX65hDg7LTbc4QgCKe0yKiTT3nnGgpHD8F7BLJzSdE3UTgTzvYGrsHo1HYX7RfqLuvaXaN9VQVUpLjrpIlIWqWRxPC9ddT/Jdz6P+MjYwwMVdgyW0hg9OOY7fRWffzPQFosQiaIjPGV3Qn6BwYiosb8QvmgfYeUAccNEa5ePlfZisW5pVxRoArlwSVtD8ymtZyqK4GJAxaqdq4HjK71sgDC70pjAp9NYY8q/h/YE6rGyMwjJLyueolE4Qdko4vbw+TV5ztQ1UlndDNVPcrD4tVRtOipiiShDU9DWsJK9/4XWBwVQSdZDeYxcdDMtyhk1TI+N9l5Vi0kpIeP6SIA8lggoVpmzbCAPMyZnLi9IfrYBqh+TTpouY7ccT47pGzh9qjywNNWdGPFmdddNJCGbANkhNviK+cwpzkXhIsXIOs9joM5MPQuG7HxVNLAnvXz8IuYhmYO2j5yThd0yosUMp7xQDs9oG13He4hx3/rN+SuDzfQ4/OGa1/hW6FG5Yg2Lm6vXnCTSoROF6bIaBspKu+Gsvc5pglBcFkoEIm7KoASD/p6+lNVu9tRnPJl84CYbjwmp5pQiPTF9eujtaFLDZUs+ZA5BhG9Ha0ytk5H/6DQLJpDZEJsLsYBivM7VK1Q8GoV9Z+w1mDdA6RIrwbk2XpGyDgrPtLUYicQwbyGs1uP0Gj7SD2fxqBh01+kaBkQLjS6gDEqvZ6Wn40sm4RIRxHaGmShNo1eEQtDhE1iEPyuJ+deacLbfUO2g45KbP4KDuBEDMHOtQ3xbobaF9QhkmjjJd/4/ifAAiO4JpmA5M2G1k46GSuuU4XG62c4pXNgENeU8Fnf8ujuBOeQfVtOd8ld+EPs9SOUMqVnvjlXb37DViRq4kP5OefOlKIVeoIwA3Hn7dOYxOK9llXBESwTotewSp/EP9bi7tgQ7xL4Sn+mM7UP1bcjI 7vFukVC6 tUcKP6CU0UnmHdES/hJgXwvPHe3rl9AIYQVFFGtJ/DAjWFdbUG07K80e3tpDdOUB36thu41IGCDX7RnWuTZNteU7zuOqsTnvr3p1mb7WjSgtAEeM++AosavaZcFCxvI3XIiE9Tv4TTisqVryjcQ4p2hxJt0lLOYYPeMj5F2he+hV0aogf1tKba5NMG07yzlkTB7ftheD2Z+RMhbRhPD473pN4deibQ4STXxiWCUDvc5LpDdJuA0d/cTQmvw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: hugetlb rmap handling differs quite a lot from "ordinary" rmap code. For example, hugetlb currently only supports entire mappings, and treats any mapping as mapped using a single "logical PTE". Let's move it out of the way so we can overhaul our "ordinary" rmap. implementation/interface. Let's introduce and use hugetlb_remove_rmap() and remove the hugetlb code from page_remove_rmap(). This effectively removes one check on the small-folio path as well. Note: all possible candidates that need care are page_remove_rmap() that pass compound=true. Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts Reviewed-by: Matthew Wilcox (Oracle) --- include/linux/rmap.h | 5 +++++ mm/hugetlb.c | 4 ++-- mm/rmap.c | 17 ++++++++--------- 3 files changed, 15 insertions(+), 11 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 0bfea866f39b..d85bd1d4de04 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -213,6 +213,11 @@ void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); +static inline void hugetlb_remove_rmap(struct folio *folio) +{ + atomic_dec(&folio->_entire_mapcount); +} + static inline void __page_dup_rmap(struct page *page, bool compound) { if (compound) { diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 305f3ca1dee6..ef48ae673890 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5676,7 +5676,7 @@ void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct *vma, make_pte_marker(PTE_MARKER_UFFD_WP), sz); hugetlb_count_sub(pages_per_huge_page(h), mm); - page_remove_rmap(page, vma, true); + hugetlb_remove_rmap(page_folio(page)); spin_unlock(ptl); tlb_remove_page_size(tlb, page, huge_page_size(h)); @@ -5987,7 +5987,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* Break COW or unshare */ huge_ptep_clear_flush(vma, haddr, ptep); - page_remove_rmap(&old_folio->page, vma, true); + hugetlb_remove_rmap(old_folio); hugetlb_add_new_anon_rmap(new_folio, vma, haddr); if (huge_pte_uffd_wp(pte)) newpte = huge_pte_mkuffd_wp(newpte); diff --git a/mm/rmap.c b/mm/rmap.c index 80d42c31281a..4e60c1f38eaa 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1482,13 +1482,6 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, VM_BUG_ON_PAGE(compound && !PageHead(page), page); - /* Hugetlb pages are not counted in NR_*MAPPED */ - if (unlikely(folio_test_hugetlb(folio))) { - /* hugetlb pages are always mapped with pmds */ - atomic_dec(&folio->_entire_mapcount); - return; - } - /* Is page being unmapped by PTE? Is this its last map to be removed? */ if (likely(!compound)) { last = atomic_add_negative(-1, &page->_mapcount); @@ -1846,7 +1839,10 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, dec_mm_counter(mm, mm_counter_file(&folio->page)); } discard: - page_remove_rmap(subpage, vma, folio_test_hugetlb(folio)); + if (unlikely(folio_test_hugetlb(folio))) + hugetlb_remove_rmap(folio); + else + page_remove_rmap(subpage, vma, false); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); @@ -2199,7 +2195,10 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, */ } - page_remove_rmap(subpage, vma, folio_test_hugetlb(folio)); + if (unlikely(folio_test_hugetlb(folio))) + hugetlb_remove_rmap(folio); + else + page_remove_rmap(subpage, vma, false); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); From patchwork Mon Dec 11 15:56:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487437 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69D9BC46CC5 for ; Mon, 11 Dec 2023 15:57:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EB9CE6B00FE; Mon, 11 Dec 2023 10:57:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E66AA6B0100; Mon, 11 Dec 2023 10:57:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D082D6B0101; Mon, 11 Dec 2023 10:57:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id BA01C6B00FE for ; Mon, 11 Dec 2023 10:57:12 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 90B86A1BC4 for ; Mon, 11 Dec 2023 15:57:12 +0000 (UTC) X-FDA: 81554991504.23.FB04B59 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf12.hostedemail.com (Postfix) with ESMTP id D109340022 for ; Mon, 11 Dec 2023 15:57:10 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="cK/oeP1o"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310230; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eQFoT8xrLd9LUZRih2ZF29493QEHiYzpAeVB57UisD0=; b=H5OffoPXzgKH1H+SO6xIYtNpdy+oYCpdI+/1n8P+qrEayoHHnqCwKm7vXV9EUP/gcgDuK3 gdzUW7WjPdgndMI4pNqm6Hlo5JND52rhOUIdx8t7KTmSUVosQhzfzV2dZZMB1d2BDqcyXt g6g8JEd8pWeZy4o96+4EI6ENswBu/0k= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="cK/oeP1o"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310230; a=rsa-sha256; cv=none; b=bPpo1/+jtO4FHuo9l2ufMQlSWLV6T++k4Dp3gSv42Fp+N1BADQyRmqbgg5Xz+sEFnGpcMe FWf1cxhXN7gMKV2F0z/a0OceTpSiMVH22DwxjAqzHMbmZbR7MXmr3cr6ZSd7I1dYwjsAYm TJHQ06icrgdmc5XW9UwEQqqgs1AKWL0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310230; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eQFoT8xrLd9LUZRih2ZF29493QEHiYzpAeVB57UisD0=; b=cK/oeP1ouam/D+5+S4uhtaHIB7DBFVudnPTmX7aEjMpRgaClH695MluwewQAhP22ob9dVc GFPnwk9nS1q1Y7huvCaxx2lx//1oG1beysZJP1TcN6MFe1QsOAzCZrIlcwt7FSXYHMoy2j pKCjW4VFUrzdyRZFWKKxH+KecZ79IJw= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-620-xxnUSZcjMXupZYk_PaLjmg-1; Mon, 11 Dec 2023 10:57:05 -0500 X-MC-Unique: xxnUSZcjMXupZYk_PaLjmg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 8D0001C01B3A; Mon, 11 Dec 2023 15:57:04 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2EA5E1121306; Mon, 11 Dec 2023 15:57:01 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 03/39] mm/rmap: introduce and use hugetlb_add_file_rmap() Date: Mon, 11 Dec 2023 16:56:16 +0100 Message-ID: <20231211155652.131054-4-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: D109340022 X-Stat-Signature: mmi6aurns4c8yfbsgierg4ojcnhweni4 X-HE-Tag: 1702310230-354486 X-HE-Meta: U2FsdGVkX198MTYIo6tCsuxIBq/S6S+7KCMRfi4THdZettZks9Bim7sckNTnMfWjngWE+x92//AeLypVQtQmjuqJcyPIODPPHmwLFir5kguNUKsz4C2hd/wAEDjMiKBbSeywfbdCiaTaPz+J/uX8CalqgCKgeURUlQgONLL6uhv9BLr+WBDrC8Mpwh3+qvFSppXiugBecPrEZYlAsIK1pPFb7rLWDYntAf8fHWjx3n1/wozOukzRqEl3hJYpzKEbTBfLg8yMozflzXuegytXLkfGO+gM43ixUNB8NLskxlLXOWH3+V6HBZgeqybP0dwnkgSGs/HFxzzeHcd2O27OP+v512HKqCv9J8aYTCh9zNTnQswuAfIKQc0lQzW1RckAhXu1D8O3WOy8UrJYnnlEYW+L+/rIxRZ2O13RwgCaHLu3SS2/9QvwEdKh5rvkOIBLsm3gEeAbsb4LzYCF6Bo03PQtGQsPcSs4ZyAKVI6OpY3rX9p7fE6NJzoQWNFz8m9CwzVYbLM115rqk7+6ubwHhsGjzjlKuyXDK0zfmwLYkQSfUhAQ+79r1qPAz+9Tto9uNBhCP9//W964yh5VigF4cnhTXi/ioontiuAaFt/3p9jNZOphYuN9UhfOmfs0vxE5q1V75gM2iqPGDbWfVinzvQEYhZIr1pZSA7R0T4y0cOwky578bpkd4GiDk2cUhwnR8sySNgoamxVpHmonizsira/ZSURC/MdSQFqWMBogQ6QcvN4+fUl8wdzngN1CNaAKiuo2mKGPQRmDojLfnAgzP/cTNhqKDj73pNaVMY5kEkgsEXW5oRMPBotRVv6mKqYU/1UqfgPvxAYdqfmXtaJt/+BwrJC0nn3k9/IgWRf+YRSuPdJ+v4kNTvxvaA5xV8uAsEJwrwWoE7LZYiNSyt5u6Y5qd2XkFP2MuEOT6UxXUYnEzxps+jURGxD4xJw2YDgrikHmPl64Txc+Ccw7H6W SkSjUTC8 Xryn0NhMG1f52mFTlMrstX0oJx8pBwuErMjzL/qCxtuq7omHJNaY1ONhM1kU0vh8FWxs9gVgAPG63YehmxgTojGVLCYwEdgikrmRFmadrq6pMdVzF/7FeVzeaHv19wx9oVQ6f1jqVII7T45Ed4xmgFk5dm+QPW+7XtuCHfleA0QYotTepT44Y3GdpNNVRArRJEey9bpwoXE5YZ/DmMENRjIXqEXT+77SknHjJbILlPMFEi+v0+JNP0I3krg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: hugetlb rmap handling differs quite a lot from "ordinary" rmap code. For example, hugetlb currently only supports entire mappings, and treats any mapping as mapped using a single "logical PTE". Let's move it out of the way so we can overhaul our "ordinary" rmap. implementation/interface. Right now we're using page_dup_file_rmap() in some cases where "ordinary" rmap code would have used page_add_file_rmap(). So let's introduce and use hugetlb_add_file_rmap() instead. We won't be adding a "hugetlb_dup_file_rmap()" functon for the fork() case, as it would be doing the same: "dup" is just an optimization for "add". What remains is a single page_dup_file_rmap() call in fork() code. Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- include/linux/rmap.h | 7 +++++++ mm/hugetlb.c | 6 +++--- mm/migrate.c | 2 +- 3 files changed, 11 insertions(+), 4 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index d85bd1d4de04..91178d1aa028 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -213,6 +213,13 @@ void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); +static inline void hugetlb_add_file_rmap(struct folio *folio) +{ + VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); + + atomic_inc(&folio->_entire_mapcount); +} + static inline void hugetlb_remove_rmap(struct folio *folio) { atomic_dec(&folio->_entire_mapcount); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index ef48ae673890..57e898187931 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5408,7 +5408,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, * sleep during the process. */ if (!folio_test_anon(pte_folio)) { - page_dup_file_rmap(&pte_folio->page, true); + hugetlb_add_file_rmap(pte_folio); } else if (page_try_dup_anon_rmap(&pte_folio->page, true, src_vma)) { pte_t src_pte_old = entry; @@ -6279,7 +6279,7 @@ static vm_fault_t hugetlb_no_page(struct mm_struct *mm, if (anon_rmap) hugetlb_add_new_anon_rmap(folio, vma, haddr); else - page_dup_file_rmap(&folio->page, true); + hugetlb_add_file_rmap(folio); new_pte = make_huge_pte(vma, &folio->page, ((vma->vm_flags & VM_WRITE) && (vma->vm_flags & VM_SHARED))); /* @@ -6730,7 +6730,7 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, goto out_release_unlock; if (folio_in_pagecache) - page_dup_file_rmap(&folio->page, true); + hugetlb_add_file_rmap(folio); else hugetlb_add_new_anon_rmap(folio, dst_vma, dst_addr); diff --git a/mm/migrate.c b/mm/migrate.c index 4cb849fa0dd2..de9d94b99ab7 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -252,7 +252,7 @@ static bool remove_migration_pte(struct folio *folio, hugetlb_add_anon_rmap(folio, vma, pvmw.address, rmap_flags); else - page_dup_file_rmap(new, true); + hugetlb_add_file_rmap(folio); set_huge_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte, psize); } else From patchwork Mon Dec 11 15:56:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487443 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 464F1C4167B for ; Mon, 11 Dec 2023 15:57:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 20FC96B010B; Mon, 11 Dec 2023 10:57:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 199516B010C; Mon, 11 Dec 2023 10:57:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F043D6B010D; Mon, 11 Dec 2023 10:57:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D551C6B010B for ; Mon, 11 Dec 2023 10:57:29 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B28AFC072F for ; Mon, 11 Dec 2023 15:57:29 +0000 (UTC) X-FDA: 81554992218.06.27EAFFF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf18.hostedemail.com (Postfix) with ESMTP id EB1861C000E for ; Mon, 11 Dec 2023 15:57:27 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Ls/z1JqL"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310248; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=U+sz0aAg3eDeTtLeKtB884YEpFTON3wSVy4B1/EJUUg=; b=kIEavdMPDO/BWfLpsAnjfqVTB1/HJNTVkESEYyH2jxpZe0ibqUNmOq+5xCNEXCfGuFbpZe w2nEqVYnPl44zTkmT4Kvae8c5Ur9wyEomXMl0NDFGzIscAvKaKiF9dpBtNevx1dwWRmkRy TrDqYT7mJSbWQ0JmR/5CTIn2AoSRKks= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Ls/z1JqL"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310248; a=rsa-sha256; cv=none; b=B0Glj5BFaNimnkyKYiU7HYGlcM6NRoCdfbPkBmsN7J/BNGKtTJasLFXLpNLW/Q6DkdHKgY 2iJ/6QZeNBJF612PMeLLLc+ijsQxc/R5nbo4WiFFmA9tYDKpjLFKQUZWVOnbtHPgFkaslJ 06B5XA8r+hWnTWU+g50NIAyne3mthiY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310247; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=U+sz0aAg3eDeTtLeKtB884YEpFTON3wSVy4B1/EJUUg=; b=Ls/z1JqLgi0iMIlyc5ggOIWWJgrZ+VxJFFfpiQzdWr/WgwPINDQrAPvWO7W7xxKDnPS/Is btqAmg18cBXXAVoHk9tMe/MnSFo15RRxHaIrSMQ07DGrDDSzdbMG3xwUY+yoS6enitoOZJ qjQ2TukFvN8WJbd+mrPWxIkeZdOLIak= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-509-x5EZWnqnPL-8WFVv6DDwvA-1; Mon, 11 Dec 2023 10:57:23 -0500 X-MC-Unique: x5EZWnqnPL-8WFVv6DDwvA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9899D101E156; Mon, 11 Dec 2023 15:57:07 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id EAAC51121306; Mon, 11 Dec 2023 15:57:04 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 04/39] mm/rmap: introduce and use hugetlb_try_dup_anon_rmap() Date: Mon, 11 Dec 2023 16:56:17 +0100 Message-ID: <20231211155652.131054-5-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: EB1861C000E X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 7519ofrj93wqhr8cmw6z31m7fnffomn5 X-HE-Tag: 1702310247-615679 X-HE-Meta: U2FsdGVkX19zY0GPsKoc1wmQlpp6Y792bX1Isrk86tsUI9lse/fWxNCh1Kq1b/quXZ0r2GkTTXGgwmoijeVaz+IdwzY5qf/CLyyVi8h16FAnmYNcq9653xuUtBneqGLZFaPCATPGquG6hAxrTSbIqW03P0v6N9BxgNGleqSKR86nknK7fUN+WiP3UzcZO3N2VYxVzHYRM81K1fXDCuvP7jT/NS4NUINZwHMRUEtaDJ9rnoA75ScQnJf3/qjxqp+fBQE84xw2mw2aoU/qaFHBYCGDGS7YWzPRtnxb/qbnwawSDgUY1vObeGuSnf/GEChHmXSG+zj9nCYc5iqBQ/VcgNTMuBWSJ9qMd+vkk6sYYdaERapTpjsCIrIbmHI9OY5081H2ilq295OPklmG1cTtKJ4ut/deksFpIBaPKPpMYfe98K6VceN2VYS1vkbB+H1iESt2Unt5gUM6P4C3/tYaqU3Kq5thDjw5UJ9epeyIor5i0IATGz86HKEpwjx8MFvpbZ9y2LcL9OuMldUEHuUh7ewkvZT9AebVyVHNEDw0fB591Ds8tS23AGoIaeAhNn9md+JUE3mcZ7UNFsopOmpb3k89OOdMTXkXF64PS4Xwkezw7RRDPTgriYH+FjlWlCSi21FteqccqXs4zgXgy+hv37y01dURpRWGUuLYcT7x3092kDRzCra/6aWEkIa+9BP+Ut6mtW2nSZZFgk6QWNiuuckkh9YMeuLlh/Mrrnmy8hvx5ZhPMjJC5pDtxpuel+fzdwa0r2eNzH9tUIhPw9iESzsBjKBDGBMQRKRi4kowuyHuUYqlRT9MGrvWLgSxmjVPwRwmP0+gOyDHYqd7Efkro79vmif0Sdsk2iD+gGEpYH8l7iYu1NMrfAUWllqCUOGUkCUigRNYpR+BgmuaUD6h6Chwq5olzO8fRMqdBVkm+ja9r6mAIK7ORl++5JOcqofmqZqZMAMjIWWC8AJf3+E Hwk7r25R dsxmEMplRy+V1GWPwDfpSzh7nhJve5D6Kqw9qQ6LtaIZquMWZqYF5bcBESma1o3oCpJmUTFLTmm9MVtbuhL/YxF7O7rQoBugR2qlkEmzOIARXErE+D34BpuLM+RCVcADxk1CbXJs8aPmxto/fYVY/6cpbaxV6qEAbqcDx0h8r0fNBzGVTWCVse79pxDa/rlQy+/ivZ0iZHRx5p/7OSI+iJb803hlissbi7CNWwNqDTrokQL9bP63tlmeGLQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: hugetlb rmap handling differs quite a lot from "ordinary" rmap code. For example, hugetlb currently only supports entire mappings, and treats any mapping as mapped using a single "logical PTE". Let's move it out of the way so we can overhaul our "ordinary" rmap. implementation/interface. So let's introduce and use hugetlb_try_dup_anon_rmap() to make all hugetlb handling use dedicated hugetlb_* rmap functions. Note that is_device_private_page() does not apply to hugetlb. Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- include/linux/mm.h | 12 +++++++++--- include/linux/rmap.h | 15 +++++++++++++++ mm/hugetlb.c | 3 +-- 3 files changed, 25 insertions(+), 5 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index b72bf25a45cf..ae547b62f325 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1964,15 +1964,21 @@ static inline bool page_maybe_dma_pinned(struct page *page) * * The caller has to hold the PT lock and the vma->vm_mm->->write_protect_seq. */ -static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma, - struct page *page) +static inline bool folio_needs_cow_for_dma(struct vm_area_struct *vma, + struct folio *folio) { VM_BUG_ON(!(raw_read_seqcount(&vma->vm_mm->write_protect_seq) & 1)); if (!test_bit(MMF_HAS_PINNED, &vma->vm_mm->flags)) return false; - return page_maybe_dma_pinned(page); + return folio_maybe_dma_pinned(folio); +} + +static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma, + struct page *page) +{ + return folio_needs_cow_for_dma(vma, page_folio(page)); } /** diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 91178d1aa028..ca42b3db5688 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -213,6 +213,21 @@ void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); +/* See page_try_dup_anon_rmap() */ +static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, + struct vm_area_struct *vma) +{ + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); + + if (PageAnonExclusive(&folio->page)) { + if (unlikely(folio_needs_cow_for_dma(vma, folio))) + return -EBUSY; + ClearPageAnonExclusive(&folio->page); + } + atomic_inc(&folio->_entire_mapcount); + return 0; +} + static inline void hugetlb_add_file_rmap(struct folio *folio) { VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 57e898187931..378e460a6ab4 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5409,8 +5409,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, */ if (!folio_test_anon(pte_folio)) { hugetlb_add_file_rmap(pte_folio); - } else if (page_try_dup_anon_rmap(&pte_folio->page, - true, src_vma)) { + } else if (hugetlb_try_dup_anon_rmap(pte_folio, src_vma)) { pte_t src_pte_old = entry; struct folio *new_folio; From patchwork Mon Dec 11 15:56:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487438 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 295B8C4167B for ; Mon, 11 Dec 2023 15:57:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B34626B0100; Mon, 11 Dec 2023 10:57:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AE4296B0102; Mon, 11 Dec 2023 10:57:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9AD506B0103; Mon, 11 Dec 2023 10:57:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 883C26B0100 for ; Mon, 11 Dec 2023 10:57:18 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 39B03120790 for ; Mon, 11 Dec 2023 15:57:18 +0000 (UTC) X-FDA: 81554991756.27.243CEBF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 710C040009 for ; Mon, 11 Dec 2023 15:57:16 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bEMFaQXf; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310236; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NEVRdWv5MmhzBcZ7w5tNkRp7uojKujUjmq26NniU0lw=; b=y27BZaQgmOJOXw2ZVppwbjMwiIZ/BeWwpk+U1QrYpUpNMmmLyjjMbpSKTxbcR71QM10flR 6U+Xz/wepn0r3lljdmmPwpbpduAJm8pZZfIcxDky3HdhloCB2/dpJT/AkPf7U5tFRrgZWv AW5R8RqyE/uY3tmstRU6u0zGpnDW3qA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310236; a=rsa-sha256; cv=none; b=6By/6zpjlMdsVmwmuDscmAUnLjBG5DKJHOpsSTQ89MtmbwEBHZsH6KcO71oUURLmHLm+1R 8+tRv5juiz37HmZzFJqqoFuxdaH/1GQhKQ2TyXtnj+VCwMeOK2I4ocMYV2qiZU2Aigdh7j 3H2hiKHEH/c/jgGsOPOhsbcw+efJM+k= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bEMFaQXf; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310235; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NEVRdWv5MmhzBcZ7w5tNkRp7uojKujUjmq26NniU0lw=; b=bEMFaQXfk0hztuuzstaaJvONUWUpQgUpBaijog4k4rH4OtdQTi4AMKwa+xzwuCEKreJVvR 1FyCLgS5Rz85I1SDOQ+0n/YDbXe3HXnngsZ2YGDc+DEJCJF80MUttMVC5x/JYXOECsUwD5 q+VmeCIylVb/jts0COGyhPB3cCUL4M0= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-173-qBz2N7eWPmuxhrmCuHZ-Dw-1; Mon, 11 Dec 2023 10:57:12 -0500 X-MC-Unique: qBz2N7eWPmuxhrmCuHZ-Dw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3226A28078D6; Mon, 11 Dec 2023 15:57:10 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 071301121306; Mon, 11 Dec 2023 15:57:07 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 05/39] mm/rmap: introduce and use hugetlb_try_share_anon_rmap() Date: Mon, 11 Dec 2023 16:56:18 +0100 Message-ID: <20231211155652.131054-6-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Stat-Signature: 3psrtjt3ck1zse59oa9qp4zdgwqx4xpo X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 710C040009 X-Rspam-User: X-HE-Tag: 1702310236-287143 X-HE-Meta: U2FsdGVkX1/D2aN5vnBeKkG9d9FiiGKngYgBtdAFIaN0RTRT2mRC4NGBbPp5anY+IRFNLg6+B6oCQCDuDscsPmS4/t7LnJrBtDnx5hinyOfYtBYAn95jwJIr+ftG+n+9XAuz3UVJZXcEnAVyQe6YIyz3Qhrsj0cb1+d/6Da9p4z1E3GYk0zhanbyhCooXSzc9oRpUnZTu74RynHVt4HB8TLUUg2fbuetOneTnXtOdiUXB+ZnUZ/KzIFWB4RHhDgdlBKaDZITvTFjP0OBLY1BI8MQhLqOGs7FB5lrsc2La9fW2kZnus/+TM8g4zUnZ5YH8pWkDdDaKqc1F+fygK/26yxXIaggztE0q0FYGP5F53wcYOiiHTSJgA+5PklN/bG+N8iYG7U8FaEGAXxc8w/9QVXEAzQL7K/1fXd17Do8aoC1YNAhqb8VG4G/n5A9inmadUg8gkLijrIz7qwiREBmDJF+o1NYNhfv1wDtUGRZDzpSmPUGpPwSKPW/Kt/RWr4PoCYXlwGpFHSw1xki85k4SuNxsslxuJT0+c9VCCl9AxzQxzXf0kqHbp88qpxAqwelZXpc1Ipe042nmSdYiQPoyHPHL0WmCJNvSB4URAXjervoiDhc2op7rx0h+K9dojj1bJNbttGhP8roNyhGVp5DOP7LGS7PPfSD0mCFZZ7xUL0UrsTXknlpFmZOILG6CNanMmmJDB/wUCySoBXB6fXBvsk+rBH5fT0/LP7JygRxDd7uZe70TWlFxgHYd781e43AewVMk88GX+YWt37gyX0CbwXeQEVvnC3Nf8CG8d529Y8bchNRaixjkYCisWExT86qSRdt1X5V2o4iNukuzxNl60rLPVkjxfUNDpD5jx7KnTQdWZur4iwB+b6m3koV+EVCsgC0phyDWCUoEIEilswju3nTewtVNJP11iUNY4LeaLBBYKMkqSa1WbTiVShDqI7j3UU5XfqEGnluFH+rAqj H5zyfUY4 J9VSPuk1JKU13HIlV0oEcmvWlEJDWVeKYOdyuuEzcALRBWY88a07r1KXck3vTY5VA3uyyZVRcCjJPxej4A8ksV4BronKwMgBRpac8QVpdylVIx9kLtluALxZwyd0yOXm4UzWz9tCCAfYYD5alsBFWgXmqKn6itXS3HfaCWkoUcmFRWFRziT6bRdWuhSyivjgP0P6L1wwCr6fVv65OsDYcGKGbjOIFviGnGAQBRH+PTWt3pzKDUG1hXOClmw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: hugetlb rmap handling differs quite a lot from "ordinary" rmap code. For example, hugetlb currently only supports entire mappings, and treats any mapping as mapped using a single "logical PTE". Let's move it out of the way so we can overhaul our "ordinary" rmap. implementation/interface. So let's introduce and use hugetlb_try_dup_anon_rmap() to make all hugetlb handling use dedicated hugetlb_* rmap functions. Note that try_to_unmap_one() does not need care. Easy to spot because among all that nasty hugetlb special-casing in that function, we're not using set_huge_pte_at() on the anon path -- well, and that code assumes that we would want to swapout. Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- include/linux/rmap.h | 23 +++++++++++++++++++++++ mm/rmap.c | 15 ++++++++++----- 2 files changed, 33 insertions(+), 5 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index ca42b3db5688..4c0650e9f6db 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -228,6 +228,29 @@ static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, return 0; } +/* See page_try_share_anon_rmap() */ +static inline int hugetlb_try_share_anon_rmap(struct folio *folio) +{ + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); + VM_WARN_ON_FOLIO(!PageAnonExclusive(&folio->page), folio); + + /* Paired with the memory barrier in try_grab_folio(). */ + if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) + smp_mb(); + + if (unlikely(folio_maybe_dma_pinned(folio))) + return -EBUSY; + ClearPageAnonExclusive(&folio->page); + + /* + * This is conceptually a smp_wmb() paired with the smp_rmb() in + * gup_must_unshare(). + */ + if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) + smp_mb__after_atomic(); + return 0; +} + static inline void hugetlb_add_file_rmap(struct folio *folio) { VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); diff --git a/mm/rmap.c b/mm/rmap.c index 4e60c1f38eaa..e210ac1b73de 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2147,13 +2147,18 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, !anon_exclusive, subpage); /* See page_try_share_anon_rmap(): clear PTE first. */ - if (anon_exclusive && - page_try_share_anon_rmap(subpage)) { - if (folio_test_hugetlb(folio)) + if (folio_test_hugetlb(folio)) { + if (anon_exclusive && + hugetlb_try_share_anon_rmap(folio)) { set_huge_pte_at(mm, address, pvmw.pte, pteval, hsz); - else - set_pte_at(mm, address, pvmw.pte, pteval); + ret = false; + page_vma_mapped_walk_done(&pvmw); + break; + } + } else if (anon_exclusive && + page_try_share_anon_rmap(subpage)) { + set_pte_at(mm, address, pvmw.pte, pteval); ret = false; page_vma_mapped_walk_done(&pvmw); break; From patchwork Mon Dec 11 15:56:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487439 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16416C4167B for ; Mon, 11 Dec 2023 15:57:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9AF496B0102; Mon, 11 Dec 2023 10:57:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 95F7D6B0104; Mon, 11 Dec 2023 10:57:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 826B96B0105; Mon, 11 Dec 2023 10:57:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6A1696B0102 for ; Mon, 11 Dec 2023 10:57:21 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 45D581C107A for ; Mon, 11 Dec 2023 15:57:21 +0000 (UTC) X-FDA: 81554991882.02.A779FBD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 8167A14001C for ; Mon, 11 Dec 2023 15:57:19 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VEhML5gb; spf=pass (imf26.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310239; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lrIUJZ6x1JeaI9Xj6nWMq9GTptSO/huW1R5CZYvfzvo=; b=Abctx99FHz63u0uE1h8QJmWMTTN3gj8Dqrwsfp8v/3c3+fjJr+oE3WtUYkKSA7t3P6hmxN lW99GhokHBwIX6G7OpX1npjRwwpk0tOECoVly3tfLqC9XfXmhoCOt+5QeMwCbjzL6BGcZV +9eX8tG0P7dHC+IiWZxYmbF/Lu3W/oE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310239; a=rsa-sha256; cv=none; b=yjyBHGuwdQuyOE14KEFiIL6qbTtn13WOYrxtuEhNihPfUn/V9KWWGrfPze1b8psmog4MeH wn9qPFNjEwWpf3x9Rg2ihRe8Xy75UUO8rN3VmsaIzljbOJ6cqnPMiGx+2PM2c51n47byCm e5HLzrbuFjsOjm9HeAmuVVFspG6D2oE= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VEhML5gb; spf=pass (imf26.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310238; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lrIUJZ6x1JeaI9Xj6nWMq9GTptSO/huW1R5CZYvfzvo=; b=VEhML5gbi7xgejf952AFYfm9bz92LbbKF/9EF/RFvpUXXNkfUq5RuW9ceBMqPbWxFZiG6o A5tuAo3zfpAvBWrI8LTkIcV1xr4a/CVwY2xt8BWC1rfT+ZBELkcnjBB9/D5By6PiG3XooR cYiS3m9RVSB4U8tnQ04+4/zoLsv+uS0= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-213-1tzqxM8YMK2PmJTvvwdWZQ-1; Mon, 11 Dec 2023 10:57:13 -0500 X-MC-Unique: 1tzqxM8YMK2PmJTvvwdWZQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 864033C28644; Mon, 11 Dec 2023 15:57:12 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6A5831121306; Mon, 11 Dec 2023 15:57:10 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 06/39] mm/rmap: add hugetlb sanity checks Date: Mon, 11 Dec 2023 16:56:19 +0100 Message-ID: <20231211155652.131054-7-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Stat-Signature: y68o4wjg6ki8oyibx67f79eb95q4b54p X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 8167A14001C X-Rspam-User: X-HE-Tag: 1702310239-196160 X-HE-Meta: U2FsdGVkX19dmI8lxHQ6ZqIs0uuDpi/okDx3BdsRI/weQTNo1gn8qnqAXPvQfWuam+2hfoOWT507c/ipyYXzeHJPcuL2CwC6In8lfoOTTuzO0wV3W+Sw6C1u+6ZODSx/YNYurhF6Ym56Zisfh+gVsnH2oxJaHUl6JJu7AzS7fJ1kXahgU7mLaSSZ9fB6ue8LR9ebf4OXtSyhzu1Zl4oeRO7A/uN71QKx8v8+mxNvF1yxG1lHGPefz4l7hENhlMENKKub1ABO7ADm2iYsL47FCBE4K28R8E/sAzMmYgo4WQj0H5rI0sIp4PMfZPO4abMpc6Lm5fUxUhrqRZDh3b+VjQJ3DsvEns1AC3tfy5YTujNq1nEoCFGc0YA4nuriMxyYslGQoTn+Sg+HZTGh6W4ZAox5+/9r/xT9aQHPBeWZWnOiVileRJkGd63Ca9kAiIwoQluQ+rb14NnZuE5wtOviFDr7OvMAV9Uul4ZZ/u8oSHkUHrDcXU7pXnJqECKfnyL260stCpnSTpTLVyJ4Q9xOgPgtETn6oiEKa5Q6eMFnyZZ9WYFkr6q98af9c2TzmneLyWLqitfH7J7n+1ijyn9wJcAnUSjSyjRNhgjrWoZrtO8s2lK1FOIA5Zg8M8ej5LlECf4IjIiigT5urCZE46RZNrbLu9fxJt0KLRG44wuYf9IiOjI7CpW1bOJB66gatOvAI0Z9GlOH/XB9L+98ssV2Cby+ZurTp+b8bkcSBmxRLDAep9MuKndK5FBdLqx2e448HkfC6p6oE9vm1hxz0MVzXLV4aukToghTZ9vqbvvGouoKQEsVBDeY/AegZ90R9otm+A4fOV9ARwvSlj8gvfFbWqL1rRNbJRW3cbOyDxfMb3GWKcNK3VLZnkcFBKpEohALQrU/r0mmea+KRBlJMWxXfYMQvV+1RnBTeRPG3CpYeJ5B5jkDJJ5hllXdMLwvQbfv4/B9XBALjiYqYb7CjTb djDxpVVN FLHWiJAjsrEfuoP9Bl8l90ofzfSVVlgCbcgX0k7MAvs2/nsJzTfzQT26fz/bygMchAbWkDEN0115HmTNYHUA4EttkUVaTGv2aMhOxGp0V508I3fgvlZA6i9u5pcI2+P4SIJ/GBszzFyepYFX3gT26pzZ4DVc80T6eLBcicSTuaZfzQyo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's make sure we end up with the right folios in the right functions. Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- include/linux/rmap.h | 7 +++++++ mm/rmap.c | 6 ++++++ 2 files changed, 13 insertions(+) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 4c0650e9f6db..e3857d26b944 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -217,6 +217,7 @@ void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, struct vm_area_struct *vma) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); if (PageAnonExclusive(&folio->page)) { @@ -231,6 +232,7 @@ static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, /* See page_try_share_anon_rmap() */ static inline int hugetlb_try_share_anon_rmap(struct folio *folio) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); VM_WARN_ON_FOLIO(!PageAnonExclusive(&folio->page), folio); @@ -253,6 +255,7 @@ static inline int hugetlb_try_share_anon_rmap(struct folio *folio) static inline void hugetlb_add_file_rmap(struct folio *folio) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); atomic_inc(&folio->_entire_mapcount); @@ -260,11 +263,15 @@ static inline void hugetlb_add_file_rmap(struct folio *folio) static inline void hugetlb_remove_rmap(struct folio *folio) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); + atomic_dec(&folio->_entire_mapcount); } static inline void __page_dup_rmap(struct page *page, bool compound) { + VM_WARN_ON(folio_test_hugetlb(page_folio(page))); + if (compound) { struct folio *folio = (struct folio *)page; diff --git a/mm/rmap.c b/mm/rmap.c index e210ac1b73de..41597da14f26 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1343,6 +1343,7 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, { int nr = folio_nr_pages(folio); + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); VM_BUG_ON_VMA(address < vma->vm_start || address + (nr << PAGE_SHIFT) > vma->vm_end, vma); __folio_set_swapbacked(folio); @@ -1395,6 +1396,7 @@ void folio_add_file_rmap_range(struct folio *folio, struct page *page, unsigned int nr_pmdmapped = 0, first; int nr = 0; + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(compound && !folio_test_pmd_mappable(folio), folio); /* Is page being mapped by PTE? Is this its first map to be added? */ @@ -1480,6 +1482,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, bool last; enum node_stat_item idx; + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); VM_BUG_ON_PAGE(compound && !PageHead(page), page); /* Is page being unmapped by PTE? Is this its last map to be removed? */ @@ -2632,6 +2635,7 @@ void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc) void hugetlb_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, unsigned long address, rmap_t flags) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); atomic_inc(&folio->_entire_mapcount); @@ -2644,6 +2648,8 @@ void hugetlb_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, void hugetlb_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, unsigned long address) { + VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); + BUG_ON(address < vma->vm_start || address >= vma->vm_end); /* increment count (starts at -1) */ atomic_set(&folio->_entire_mapcount, 0); From patchwork Mon Dec 11 15:56:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487440 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4D52C10F13 for ; Mon, 11 Dec 2023 15:57:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 013E06B0104; Mon, 11 Dec 2023 10:57:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EDCA26B0106; Mon, 11 Dec 2023 10:57:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D7C526B0107; Mon, 11 Dec 2023 10:57:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C4D476B0104 for ; Mon, 11 Dec 2023 10:57:21 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 864E480783 for ; Mon, 11 Dec 2023 15:57:21 +0000 (UTC) X-FDA: 81554991882.27.EE8894C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf18.hostedemail.com (Postfix) with ESMTP id A31DE1C0016 for ; Mon, 11 Dec 2023 15:57:19 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RvbQmc0i; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310239; a=rsa-sha256; cv=none; b=H81bU+rk6mnyI/JejjqoAjJR5JO689S0VJW0RIRx5xRXYeZ7QHmA1is7XqdLlzSf4mgzRY yB3upAy4w8ycgx2qStd0R1oRKbhIypuAcuB5ExLBBDjBzKu+sI2o4vuN8Pry+MwrLp2XmU WzcAFfnx1TbQS7HVmCxWJT5WYk84mPI= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RvbQmc0i; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310239; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IYkHAajZElCwYCEwnUcbPY1wmlYkOiWYX0LP3vDIRKY=; b=G62ASeKRz22Fj1Ptfw0U95uPhUVsABiVEYwv53vQ78PNWEyepIl69MTOsxz0+/y/6kOWTe i1CCuGNgGZskXpEn99kwmmeWiMpfZHv3zf+fiEBnjPWJv8P9vsNZig6w01E0iMu8cdd0Wg y4Jy+SvlkE65XUywqUXnWhliBPdwfco= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310238; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IYkHAajZElCwYCEwnUcbPY1wmlYkOiWYX0LP3vDIRKY=; b=RvbQmc0ik3ioYZEe6XUX7Uf9TejbefOUuI6IS6W6G1hkbWXKYAsRw8heidaazCcH2pOV5i 3PHR07z3n/ZLcKZ2n8JgSMImKbNzIPr8kBSwFdwfjYbCZ/Z2uVAApHpEwbdVr5/MocEfta f13anXwbnxjzZ24KT6FfVLuBozQgUWI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-389-vLDkjjTeNrqXiJAcMx9kjQ-1; Mon, 11 Dec 2023 10:57:15 -0500 X-MC-Unique: vLDkjjTeNrqXiJAcMx9kjQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E97A4185A780; Mon, 11 Dec 2023 15:57:14 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id CEE511121306; Mon, 11 Dec 2023 15:57:12 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 07/39] mm/rmap: convert folio_add_file_rmap_range() into folio_add_file_rmap_[pte|ptes|pmd]() Date: Mon, 11 Dec 2023 16:56:20 +0100 Message-ID: <20231211155652.131054-8-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: A31DE1C0016 X-Stat-Signature: 5fpanxibzp3bsijd3ynhybj88fwbs5ni X-HE-Tag: 1702310239-449440 X-HE-Meta: U2FsdGVkX18oxZ7235FYePGvNW7+uGrMgu0L2oNReA4Q4c6IrefGCMc7bB38nyG0RM2zZsNFgIVEVlftz3P1hSogsPo2Pwbupb8xWdkZ7APz0lGmwYM1ejoeeCXA3w5vL2K+4nps701iKgiGq2VNuqBgWFzNeQYlYbaZy4qmun/Jza3FIFrrdlZkBtTZJ2GWX6C5AM6HAAa3wFR5WD3BW0TNFHgq7Nnv64Sn5BUveOh1MbtWwmlDjwKWlQi2tbiwP0SR4peXXMWGbM7BvC4oyRppaQeYJFfo8hrX1RRM7EY89xwOqCqtDDntJJ2p66CBHvRthGBhQ65Hs0KwPIE3o7plgQkx7ctDNTN571C9zHoy4aFxuNn6KMwiM3v1gj0ojna1I3tbo59BOiFHKX6uDpqq29afCDkpLltoeLMWvmsmC1rO1PXCz7BnYk/TCEG4fx4GHXJiv58BdHd/FwPumarl9rwbtqb5uI1z+o2aWLDwULdz3cqvobdmjrXnUBokiZ1SkZxcaCVBEAs6kSxGdcAyBvJIaoKaxIHIysGoxi9AviqhKXIeAE81Or+Sdg4X0ZP3TsPhieWiHXg1mu0oowxg5qOmvojFouAl6u3hgXFXLkJGUTmONQ8yU/uZGBoQwxxPWqkW/gpGN0g3B5cs77CC9EtlSAaEbpzunWAVQPCJDSKIWuh0SlFoq1TbJHt4ldw9jzhGhhGsZP1FRVo+kzFlkSJXIhPAqKjx8ihn8ADhHaGS9gGEeHmH1h8O7lpNW4JytGIF//f5yyPj5FurMG+jmPZNQlDdd8WM82q+nGTSbqOBJG799xzHRHcD9YNmUvjLdU3BP1Lql8ySXO7PDV17JTA5Zt7xfoQomhDl7AfAdGLDq0Xl1coKfPHm7TLJYxZZ/rYNEPIm6Xw1iRGMFYPiM0OpdcjYDa9gGFCkPvYrfpzjW/7rwFffMNcq1bmNB9v5UNxF4+RfwnfK2Fs CtiOR5Y/ vUWxeYQ5zO2cddEy12euqZVw538kAoeuGT/+rXmC19yqSTn4+gkE+go9HiKhOHCq+B1NQV62H78XXPLYHdx6Nm+HHA92cMvWFLmvn/BqRkmEoyt+Aa6g4Dd4nGImoWdJZZyHjE8Gl9uRSeOmu0f/ssZG82fTXZXHMGj80mXqQQjkJv+5r8Y1coATGqoPFqKGn7VdFNs3jJAKWZx5bntl7eimmts97Uezk6swm X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's get rid of the compound parameter and instead define implicitly which mappings we're adding. That is more future proof, easier to read and harder to mess up. Use an enum to express the granularity internally. Make the compiler always special-case on the granularity by using __always_inline. Replace the "compound" check by a switch-case that will be removed by the compiler completely. Add plenty of sanity checks with CONFIG_DEBUG_VM. Replace the folio_test_pmd_mappable() check by a config check in the caller and sanity checks. Convert the single user of folio_add_file_rmap_range(). This function design can later easily be extended to PUDs and to batch PMDs. Note that for now we don't support anything bigger than PMD-sized folios (as we cleanly separated hugetlb handling). Sanity checks will catch if that ever changes. Next up is removing page_remove_rmap() along with its "compound" parameter and smilarly converting all other rmap functions. Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei Reviewed-by: Ryan Roberts --- include/linux/rmap.h | 47 +++++++++++++++++++++++++-- mm/memory.c | 2 +- mm/rmap.c | 75 +++++++++++++++++++++++++++++--------------- 3 files changed, 95 insertions(+), 29 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index e3857d26b944..1753900f4aed 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -191,6 +191,45 @@ typedef int __bitwise rmap_t; */ #define RMAP_COMPOUND ((__force rmap_t)BIT(1)) +/* + * Internally, we're using an enum to specify the granularity. Usually, + * we make the compiler create specialized variants for the different + * granularity. + */ +enum rmap_mode { + RMAP_MODE_PTE = 0, + RMAP_MODE_PMD, +}; + +static inline void __folio_rmap_sanity_checks(struct folio *folio, + struct page *page, int nr_pages, enum rmap_mode mode) +{ + /* hugetlb folios are handled separately. */ + VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); + VM_WARN_ON_FOLIO(folio_test_large(folio) && + !folio_test_large_rmappable(folio), folio); + + VM_WARN_ON_ONCE(nr_pages <= 0); + VM_WARN_ON_FOLIO(page_folio(page) != folio, folio); + VM_WARN_ON_FOLIO(page_folio(page + nr_pages - 1) != folio, folio); + + switch (mode) { + case RMAP_MODE_PTE: + break; + case RMAP_MODE_PMD: + /* + * We don't support folios larger than a single PMD yet. So + * when RMAP_MODE_PMD is set, we assume that we are creating + * a single "entire" mapping of the folio. + */ + VM_WARN_ON_FOLIO(folio_nr_pages(folio) != HPAGE_PMD_NR, folio); + VM_WARN_ON_FOLIO(nr_pages != HPAGE_PMD_NR, folio); + break; + default: + VM_WARN_ON_ONCE(true); + } +} + /* * rmap interfaces called when adding or removing pte of page */ @@ -203,8 +242,12 @@ void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); void page_add_file_rmap(struct page *, struct vm_area_struct *, bool compound); -void folio_add_file_rmap_range(struct folio *, struct page *, unsigned int nr, - struct vm_area_struct *, bool compound); +void folio_add_file_rmap_ptes(struct folio *, struct page *, int nr_pages, + struct vm_area_struct *); +#define folio_add_file_rmap_pte(folio, page, vma) \ + folio_add_file_rmap_ptes(folio, page, 1, vma) +void folio_add_file_rmap_pmd(struct folio *, struct page *, + struct vm_area_struct *); void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); diff --git a/mm/memory.c b/mm/memory.c index 8f0b936b90b5..6a5540ba3c65 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4515,7 +4515,7 @@ void set_pte_range(struct vm_fault *vmf, struct folio *folio, folio_add_lru_vma(folio, vma); } else { add_mm_counter(vma->vm_mm, mm_counter_file(page), nr); - folio_add_file_rmap_range(folio, page, nr, vma, false); + folio_add_file_rmap_ptes(folio, page, nr, vma); } set_ptes(vma->vm_mm, addr, vmf->pte, entry, nr); diff --git a/mm/rmap.c b/mm/rmap.c index 41597da14f26..4f30930a1162 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1376,31 +1376,20 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, __lruvec_stat_mod_folio(folio, NR_ANON_MAPPED, nr); } -/** - * folio_add_file_rmap_range - add pte mapping to page range of a folio - * @folio: The folio to add the mapping to - * @page: The first page to add - * @nr_pages: The number of pages which will be mapped - * @vma: the vm area in which the mapping is added - * @compound: charge the page as compound or small page - * - * The page range of folio is defined by [first_page, first_page + nr_pages) - * - * The caller needs to hold the pte lock. - */ -void folio_add_file_rmap_range(struct folio *folio, struct page *page, - unsigned int nr_pages, struct vm_area_struct *vma, - bool compound) +static __always_inline void __folio_add_file_rmap(struct folio *folio, + struct page *page, int nr_pages, struct vm_area_struct *vma, + enum rmap_mode mode) { atomic_t *mapped = &folio->_nr_pages_mapped; unsigned int nr_pmdmapped = 0, first; int nr = 0; - VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); - VM_WARN_ON_FOLIO(compound && !folio_test_pmd_mappable(folio), folio); + VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); /* Is page being mapped by PTE? Is this its first map to be added? */ - if (likely(!compound)) { + switch (mode) { + case RMAP_MODE_PTE: do { first = atomic_inc_and_test(&page->_mapcount); if (first && folio_test_large(folio)) { @@ -1411,9 +1400,8 @@ void folio_add_file_rmap_range(struct folio *folio, struct page *page, if (first) nr++; } while (page++, --nr_pages > 0); - } else if (folio_test_pmd_mappable(folio)) { - /* That test is redundant: it's for safety or to optimize out */ - + break; + case RMAP_MODE_PMD: first = atomic_inc_and_test(&folio->_entire_mapcount); if (first) { nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); @@ -1428,6 +1416,7 @@ void folio_add_file_rmap_range(struct folio *folio, struct page *page, nr = 0; } } + break; } if (nr_pmdmapped) @@ -1441,6 +1430,43 @@ void folio_add_file_rmap_range(struct folio *folio, struct page *page, mlock_vma_folio(folio, vma); } +/** + * folio_add_file_rmap_ptes - add PTE mappings to a page range of a folio + * @folio: The folio to add the mappings to + * @page: The first page to add + * @nr_pages: The number of pages that will be mapped using PTEs + * @vma: The vm area in which the mappings are added + * + * The page range of the folio is defined by [page, page + nr_pages) + * + * The caller needs to hold the page table lock. + */ +void folio_add_file_rmap_ptes(struct folio *folio, struct page *page, + int nr_pages, struct vm_area_struct *vma) +{ + __folio_add_file_rmap(folio, page, nr_pages, vma, RMAP_MODE_PTE); +} + +/** + * folio_add_file_rmap_pmd - add a PMD mapping to a page range of a folio + * @folio: The folio to add the mapping to + * @page: The first page to add + * @vma: The vm area in which the mapping is added + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock. + */ +void folio_add_file_rmap_pmd(struct folio *folio, struct page *page, + struct vm_area_struct *vma) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + __folio_add_file_rmap(folio, page, HPAGE_PMD_NR, vma, RMAP_MODE_PMD); +#else + WARN_ON_ONCE(true); +#endif +} + /** * page_add_file_rmap - add pte mapping to a file page * @page: the page to add the mapping to @@ -1453,16 +1479,13 @@ void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, bool compound) { struct folio *folio = page_folio(page); - unsigned int nr_pages; VM_WARN_ON_ONCE_PAGE(compound && !PageTransHuge(page), page); if (likely(!compound)) - nr_pages = 1; + folio_add_file_rmap_pte(folio, page, vma); else - nr_pages = folio_nr_pages(folio); - - folio_add_file_rmap_range(folio, page, nr_pages, vma, compound); + folio_add_file_rmap_pmd(folio, page, vma); } /** From patchwork Mon Dec 11 15:56:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487441 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 761B4C4167B for ; Mon, 11 Dec 2023 15:57:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C95BF6B0106; Mon, 11 Dec 2023 10:57:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BD23F6B0108; Mon, 11 Dec 2023 10:57:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A75F86B0109; Mon, 11 Dec 2023 10:57:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 91DA06B0106 for ; Mon, 11 Dec 2023 10:57:23 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 68F2812078D for ; Mon, 11 Dec 2023 15:57:23 +0000 (UTC) X-FDA: 81554991966.09.C4F12E0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf13.hostedemail.com (Postfix) with ESMTP id B92BB20022 for ; Mon, 11 Dec 2023 15:57:21 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gfo2UJjd; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310241; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6j0c9ENvnxisPX95qRO4NJave2eFRdDuqpK0xgwx6DM=; b=EU1Su9Ji3Qbz8Ds7/khi3nInSZZIcAzpyUh7oGFT7V1JHz0FRuMYVtxRcgdQcj2gVLTyLy L7SZ3lOtUcPVfXGvDH7L7O0pz8Ctk6c0s6iQ2KwK+9p4DkeJBG+DvKlIZb5zltI2/OGFJm wfhqGtNbMPHRzI7h2+kFL+Cse+81oQs= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gfo2UJjd; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310241; a=rsa-sha256; cv=none; b=hBhvkxO/CoJe3z5F84MmbNzey3dlKyn/HbFy8po4eobU2yaZtf8SwqkqsyslkstnYesr4l aYVdijq8fy68K1HnwwXGWcsAjTi8mLNy6eurNv+ZAi2+n5ezro+iqZ/SetnzCUuXMRrdlJ j0ph454zff3+gQHYO/zaSTKfqVFruxg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310241; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6j0c9ENvnxisPX95qRO4NJave2eFRdDuqpK0xgwx6DM=; b=gfo2UJjdIc6+LWee59VC4khwpGS9QyOHXkCFtmc2JnP9nF4SwvEDWPUD+KO0oPB9g6pNgx ArhybpxztyvyzmSkgxkc3BuEcNO+gmluH+T6zWT0MeD4ffzPHZuZYNh+UiDOS+Wd1VEVhS IA0GUk/6UP62EkfvLoXJ3wfGTcVvG+g= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-453-i-re_ooQPI6Z0iay6LlX8A-1; Mon, 11 Dec 2023 10:57:17 -0500 X-MC-Unique: i-re_ooQPI6Z0iay6LlX8A-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 321B73C2865F; Mon, 11 Dec 2023 15:57:17 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2D4961121306; Mon, 11 Dec 2023 15:57:15 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 08/39] mm/memory: page_add_file_rmap() -> folio_add_file_rmap_[pte|pmd]() Date: Mon, 11 Dec 2023 16:56:21 +0100 Message-ID: <20231211155652.131054-9-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: B92BB20022 X-Stat-Signature: j599157o3aiipuoaoz1gyqz7wg84omdn X-Rspam-User: X-HE-Tag: 1702310241-839083 X-HE-Meta: U2FsdGVkX1+vaqVQERXQApcYm5p0b/ISaRzQMWdbCdFYyDUoHqLAp1NdRXtQ4H0RnOlsWp3b0tiJoL4FIVglkkF29UQbGJvbhLoOVCeuslUmZBAZOuYIjpHFo2QJVmDGA0b8gSzjo34U0nQe2g3fyPQMs57EdNhKzrWcsY9Xy6SDPZxXD2nn059l2DV3q732kEail1ySCs069s5EdHpej4gpd15pOjGI34jnVX9Vw7M0sDXEDIcnEl6VQtn7ezJ8VWTL6fPm5dw0Vu//6wgkBlYeUmTfUW27dwmCb/vHp3vXf5PS9EYUAvcOJZI45DAoPI5ftbl6ix4hLmzLz/QSObTnkTTVw8bdDDIdbKsdxvceL5bJzKqfhZAZsyxbU53ye+WvFUxw9B7MbZFoYw0c7quCMftEoUmjL9hOhfFs8w9hK1Wym7r34PmjPVfftL67GZD0bN/6/P3XgRRmKV6fbxsaj80+AHZr0ZrUH/9xpw0XLteB4nfuGQ/PLH91xGq4MsNzt6PNjRuT72ZC8Fa8pAFOdWTtRMqw+EviBdatZnl36/6X2PI5eYFgB6zbCXdgx6o/vINUerGFVXKZEIyXrBO/Xl0VDhaXJo9ghIF2k5bnqeXasDSurEO0LmjWuN2fNE/DxyXZt4HUn/0hZHb81Suf+HV6yX8L9E0exCwZmk9I+xW20Z8AI2fTdXmMvOXg+vv7SrxcwmSTMtBkd66pc2LVyusDgQUAKd8TYA5HoE7AefqSBD1Sjjw4DL5zIIvpjsZQ+ZpIavYgyYCNQht90a9ELvAobv8U3o8Hh7r7j/Gq1vJdWqPYcjo2KCAJnbUqi0E3fvdr3jq+4RsaSPHdZ02zRhDxPWIdkfyD2DlyGRrbMTrZyvB3xaMyiT7yv9Gs1akxg2kEbvn5GlGw+cmxoEHolY4K9uFGTHLzwKHNuYyRZr9Nz3kqMrg8vJFO4f1CsXfV/Fiv8b5CgNasRLA SuLkpj74 vpTIxEJs4Y9DRpiSHJYcf+pvgfpLbPZIPy53VluxuMXcLWZExRJXMS2xlIfTHVNRhGL62/t8CfzyaN6S1jz6XELkoeOEwI/um1Y7Jj8DOD/Q2TdoeZ/GqyMAAfNHVOQK9YEeG4CwTG4MgOlUqoEkbk0Mj1hsSsnTRIPVCdEu8fqGhhtxzkc/42HVykEF/HnU49g+vNiGrEHtt505UQS035P5NsqKbz2ENo9rg/cBuvCPxjymSc35r7SMDgw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert insert_page_into_pte_locked() and do_set_pmd(). While at it, perform some folio conversion. Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- mm/memory.c | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 6a5540ba3c65..70754fd65788 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1859,12 +1859,14 @@ static int validate_page_before_insert(struct page *page) static int insert_page_into_pte_locked(struct vm_area_struct *vma, pte_t *pte, unsigned long addr, struct page *page, pgprot_t prot) { + struct folio *folio = page_folio(page); + if (!pte_none(ptep_get(pte))) return -EBUSY; /* Ok, finally just insert the thing.. */ - get_page(page); + folio_get(folio); inc_mm_counter(vma->vm_mm, mm_counter_file(page)); - page_add_file_rmap(page, vma, false); + folio_add_file_rmap_pte(folio, page, vma); set_pte_at(vma->vm_mm, addr, pte, mk_pte(page, prot)); return 0; } @@ -4409,6 +4411,7 @@ static void deposit_prealloc_pte(struct vm_fault *vmf) vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) { + struct folio *folio = page_folio(page); struct vm_area_struct *vma = vmf->vma; bool write = vmf->flags & FAULT_FLAG_WRITE; unsigned long haddr = vmf->address & HPAGE_PMD_MASK; @@ -4418,8 +4421,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) if (!thp_vma_suitable_order(vma, haddr, PMD_ORDER)) return ret; - page = compound_head(page); - if (compound_order(page) != HPAGE_PMD_ORDER) + if (page != &folio->page || folio_order(folio) != HPAGE_PMD_ORDER) return ret; /* @@ -4428,7 +4430,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) * check. This kind of THP just can be PTE mapped. Access to * the corrupted subpage should trigger SIGBUS as expected. */ - if (unlikely(PageHasHWPoisoned(page))) + if (unlikely(folio_test_has_hwpoisoned(folio))) return ret; /* @@ -4452,7 +4454,7 @@ vm_fault_t do_set_pmd(struct vm_fault *vmf, struct page *page) entry = maybe_pmd_mkwrite(pmd_mkdirty(entry), vma); add_mm_counter(vma->vm_mm, mm_counter_file(page), HPAGE_PMD_NR); - page_add_file_rmap(page, vma, true); + folio_add_file_rmap_pmd(folio, page, vma); /* * deposit and withdraw with pmd lock held From patchwork Mon Dec 11 15:56:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487442 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18660C4167B for ; Mon, 11 Dec 2023 15:57:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 087616B0108; Mon, 11 Dec 2023 10:57:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 00FCD6B010A; Mon, 11 Dec 2023 10:57:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF3776B010B; Mon, 11 Dec 2023 10:57:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id CA5916B0108 for ; Mon, 11 Dec 2023 10:57:24 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id A8DE2A1B9C for ; Mon, 11 Dec 2023 15:57:24 +0000 (UTC) X-FDA: 81554992008.26.53990BD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id F24B64001F for ; Mon, 11 Dec 2023 15:57:22 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HZMGQLXA; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310243; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IFC4IqKosk2274JfAmIWQYXkgsQjdSCIu6Bum/spFuM=; b=AOdilNjTpzppbI8efkm91CM/WSOujhWh2yRsp6wBjL02f1xzL2kYt7kNtcpcq3qq8tILHk 3Wa0G7fFScUQLs8rWcJTMEYPRbDR4MzkUfpigK6c7jRw6Ua/HMpogSh6rQsu/vYrS6F5FA YftYKQjLb99QVs3tmHoIwP3k4OoUQfI= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HZMGQLXA; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310243; a=rsa-sha256; cv=none; b=ZGAM6Iq8zWvvofZsm+nwQUbJ6phw2j6rm2mzrspBSPFpnRWK2qqKX2X1RPbL6dIINTWG+x 9ezCcpXkF7OscqrKFR5JrXKcPjJckFrW/JUYkeuFvSq+kIm1X3LMJbiDwIOm38Giw2odoV QLcHcI3ZG4aaHFkqrqYMM1F6pPpaqrc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310242; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IFC4IqKosk2274JfAmIWQYXkgsQjdSCIu6Bum/spFuM=; b=HZMGQLXArDAJiKhMdCOVUgQj+Jqx9joqcE/6ry0leLGWn636y8gbvblOoOXHJga75u+OyZ /CyCVb/uTbYCGRPwdmeGpUdlrTMbkGYqRUbzzK15UdJiebCxRIdXKy8x7Lx00JRCthAfEG dsJjow90LsEL4gjsxwzrjJNfbQ7xqAo= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-554-HfiNUATqMv-A-RIgHEr9kQ-1; Mon, 11 Dec 2023 10:57:20 -0500 X-MC-Unique: HfiNUATqMv-A-RIgHEr9kQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B132B870830; Mon, 11 Dec 2023 15:57:19 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8B9311121306; Mon, 11 Dec 2023 15:57:17 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 09/39] mm/huge_memory: page_add_file_rmap() -> folio_add_file_rmap_pmd() Date: Mon, 11 Dec 2023 16:56:22 +0100 Message-ID: <20231211155652.131054-10-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: F24B64001F X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: oo6tq9fp15yro9xub167hypnfmh68snq X-HE-Tag: 1702310242-96546 X-HE-Meta: U2FsdGVkX19qdbXKAh+WKLtreXnJOsharoTAMWRFYFkgoljjh2W/eUwPdqdre0hLStffzuO1Sm1N4NjcSNv91qLlrDul4byrY7znNTuZH2o+vU0OvAY/Yym0IlkO1Mj++VLoZgNoLUZifFJvcUsn0h3zhjEKwZT3gCwXiVXb0G19aLhtayqTAUplZ1eumZrbdJO2JYgtmZeCfL03Kcm8x1lnF/7N43Lbfqv8vK+4V85KMxCKKDOmoa/JaP19MLv69946H09JcQSb0DGtB2iVHrRAbvIyMxYDroNvNBzR+BCsLuC+0NXJaqniGtHwrqDdi1YaTSz59cM4kHhIl980ru5kSQ0kJB5neQPE4JH9f+joihYcpCMWtIV3funKl9D9z/k28VMo9H1bQOkcMc52vzIr5BJ3NXO5X6r1zcq6eDPvoW9M5xF85Hao/HsfCJCZ1SKsQSLYGThvTZNu2NXanITQcKgrUl/AHPkIyJbRoorIDtZa2VX57mQzlFS75RuBFvTdCpqy1EwZhektD6MPZ5DA6RiYfJnIEYr0qGcoriAiICCWN/QgE+5cLXahr+bIRgkxTWPow5JiB3Y4bLeHDaszaimnQBpFrltIcacEfQ4lrTakmn3aQpX9VhTcaqLO27Vwaz+Ll7Lh3Fr2gqRaRpxvS7MouyIENNLKqJjJa8FjZD1o1TOw+vnnVj1/9WmB2aLsyAMQCIKisK4d4CIHn+ybRFzAsGt0Lz6npuJNYUBvfp1G/F8vxZlJEnuCAqucPk8eII8rexQH8zqRfKKYwbHFGYsbowRHBnXMKKU4+fmAip9I/iNBLYW5SQiMUFRGfB1fd6Nk5TbLq0suxrvE4jhDuZ4DBvF0zcWZktK9r4ww9K2/Yb0JKQlX9bsnHkJ/hvD5ergIYze9tAF+6bRO6xGfO79TZEBwoq6F94wHA6+K0jl1S1dWDrc6Hgf4NfwIo+JNIlpMMpWWF4Q3Gco uPdjWSkQ 2xeZPfWsBcEx1eXWPZ8enuv2nSLGAbCEvxCsvqy98c4TVfzsZnZsidftc+k/tPgODoOqosr6M4s/fjNFo33NjUevAROFTB1LqyysPBKVDhQEs7h4W0p+uef+d61uK5pIZuhoYTmoDdfFBrOEUrNdr4LvLk3nMCA/uqOG8dgPnssfO42oeG1q9wjflz6IzmB3bx3is03OMu51XdTr5uTN408DSS/59FpyBrdCEAaZ7ZSjaUoSuskyjRhwtTQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert remove_migration_pmd() and while at it, perform some folio conversion. Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- mm/huge_memory.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 3a387c6f18b6..1f5634b2f374 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3577,6 +3577,7 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) { + struct folio *folio = page_folio(new); struct vm_area_struct *vma = pvmw->vma; struct mm_struct *mm = vma->vm_mm; unsigned long address = pvmw->address; @@ -3588,7 +3589,7 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) return; entry = pmd_to_swp_entry(*pvmw->pmd); - get_page(new); + folio_get(folio); pmde = mk_huge_pmd(new, READ_ONCE(vma->vm_page_prot)); if (pmd_swp_soft_dirty(*pvmw->pmd)) pmde = pmd_mksoft_dirty(pmde); @@ -3599,10 +3600,10 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) if (!is_migration_entry_young(entry)) pmde = pmd_mkold(pmde); /* NOTE: this may contain setting soft-dirty on some archs */ - if (PageDirty(new) && is_migration_entry_dirty(entry)) + if (folio_test_dirty(folio) && is_migration_entry_dirty(entry)) pmde = pmd_mkdirty(pmde); - if (PageAnon(new)) { + if (folio_test_anon(folio)) { rmap_t rmap_flags = RMAP_COMPOUND; if (!is_readable_migration_entry(entry)) @@ -3610,9 +3611,9 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) page_add_anon_rmap(new, vma, haddr, rmap_flags); } else { - page_add_file_rmap(new, vma, true); + folio_add_file_rmap_pmd(folio, new, vma); } - VM_BUG_ON(pmd_write(pmde) && PageAnon(new) && !PageAnonExclusive(new)); + VM_BUG_ON(pmd_write(pmde) && folio_test_anon(folio) && !PageAnonExclusive(new)); set_pmd_at(mm, haddr, pvmw->pmd, pmde); /* No need to invalidate - it was non-present before */ From patchwork Mon Dec 11 15:56:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487480 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76BA8C4167B for ; Mon, 11 Dec 2023 15:58:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6A3946B0146; Mon, 11 Dec 2023 10:58:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 58EA16B0148; Mon, 11 Dec 2023 10:58:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 40B7A6B0149; Mon, 11 Dec 2023 10:58:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2AAD76B0146 for ; Mon, 11 Dec 2023 10:58:51 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id EE290807E3 for ; Mon, 11 Dec 2023 15:58:50 +0000 (UTC) X-FDA: 81554995620.10.6C2CC0A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 4981740002 for ; Mon, 11 Dec 2023 15:58:44 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=eaAA5Ctz; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310329; a=rsa-sha256; cv=none; b=y2jNxQVYE2Wwcv7Mktu7jqBG+UfPFUGWhTZgtIaxITjl+rdqc+PW/KnzdfUiD/rGNCCxcF Nj1OdbF6wqq6D4v2EGPeRMF+5AnrpQ9JKfUgfkIhR4vNSsPwjcEAU3lMlhe7P7YJIxJ5e7 bdFSsNxV/Jiw3Jbu4UX3gRkAAdMVQhQ= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=eaAA5Ctz; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310329; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XZDdgML9rdzFWWmlqwVPDDDqGkwGAqFc+AWaxqMx8Tw=; b=nqvWCkghFfzpHu0FKHOB+gzhNl7CL2Yzlk7isyno+d4wNCqze/k1YmhPZ6vqHUiaxCJO4T f+11IAG/2W7Sy0FxthOF6xg2q3YgUhJ6GFZ9BcIUh0QPmKmDcqrZjZrCYg6NkV0vZEE0Hq k3iU3QD1T4sa2zhEKLCmsnG5D8hsrh8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310320; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XZDdgML9rdzFWWmlqwVPDDDqGkwGAqFc+AWaxqMx8Tw=; b=eaAA5CtzNPb3B1D+YZMIi4YeSdp9ykX2KQ+bN1NYOf1uh0ezm6UfvTOSI+fYGL76BoLyE+ 3DZluI6P00uXAAT1wgd7D68x69TMEQbnhux5B3GACKwha4d4zEgkoFOXS39hfAUseMTlvQ YKwYo5aZXd/RO/37qPLPYOVaD4EO5dQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-657-X45q_OfKOJOoxzulSDa8Qg-1; Mon, 11 Dec 2023 10:58:05 -0500 X-MC-Unique: X45q_OfKOJOoxzulSDa8Qg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 57BC9101A555; Mon, 11 Dec 2023 15:57:22 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 18B491121306; Mon, 11 Dec 2023 15:57:19 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 10/39] mm/migrate: page_add_file_rmap() -> folio_add_file_rmap_pte() Date: Mon, 11 Dec 2023 16:56:23 +0100 Message-ID: <20231211155652.131054-11-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 4981740002 X-Stat-Signature: eddru1yheus3t3ndect3mc168e4ej54s X-HE-Tag: 1702310324-518883 X-HE-Meta: U2FsdGVkX19VarXTP8KDQlKMGWrqCc0QZH1ghW3c7/FJycuaoIV0d08l/yJk6x9d0B4ZX5PrUfUDOD51pxlQISUScvzIQ83RePbrYerQ5e916S5t/q69e+kX2Ox/VFuAaWodwTetXU8BID/x0OHWF2w69DdMKJpWVLL9dtw9reAKGdZk+WIWiv3NGGT9TWlRk1T7vzpyfwbMuUzyTCPbjos4MybFgQgYyT6ZTG5LopiNqPTEwkg6vZ0TSrRMWk3RmulFRvJ9zInuhaDoroliriIJndgx8qJHZ0uEdf39U7iJDLN3DsB3atUzIeb4O1XgRUn2PEBX3guRgfM/wddlR9f/m7dF9UmXy+fCxUfBCtLYCjFfnQo1bsPJAiWxQMyKfqY+SbDqJevznP7DbwTLmpg8GkokjWQv0OXF9ALclyuMtxINpmlT7Z8BK1s/F6IFzPQdDrRbzZGZhnv3Lqhhrp/FjCgVASy4SWNlHM70b9yyNGp+ib5bdnmA6JPIIrBjG6E0un0fQSILlMIx1IidJWS0rv6JsU6sykLsEjZNydiKObwWGLl1Z5tFU+JIeL/30M0SCtvw5YPgchFRBPkBLIbgOmE2iiq6RchyCOonoH5b8JhYU0FbqMlxEUUaS0ZJbRQR5lp9wH3xbrtgAVREVE7jfqnGMbQEqwr/Ox5GiS6BvRD8OnskonlsP6h8B1hSEK/KrBBBCwMfqqjtQ/dXaX8X71f2A8R5WCaNIW+xiUlpRJ6ONc409JzpxV0uQZ5VcfxPcv3E2y7b923GhgAoWKUqa+BlO5zrSZWLudbqtA77ScmMJ/qHGPFJClxfuQdsQZNx3PmpBwNtq9lKD5i8btajJgHvoH4k9wZ9v2vFrVdhw9p6UQglf2pdgFDyUkYPI2wwOCcy7SgQuf4oiSvsSE0h/FXBPq7pSg1cbNCDTB9yvs7y8FDIYt1cen7kJqL8hGA7prxfkzBzT/vVGOa 5Nb9xv/8 Q4VGN33M3cWDpA4+2RF4ByAFIl+eo+Umc0qI0IDCiEqlWHW8OL+1XcneN6/MGM9mTpSDb0zG8f6ztJ+gi2p+a2RDJgdogaFLSgn2iD2xDlwQC7rCTPhVZvBNXQrXZzX8vNLcq9AVIqxT9cGexT42u/ExrwI6UVUnuKA2gVuQlnr4/+NK9HnROi0T/04JuTGEwx4Y3mfMQ007jGOnq+Vm7kU+wuiGUb2yRa0bdaQnMNzn/Jnob3b557arQ7+YPAW82jYw/lijInuX8FeA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert remove_migration_pte(). Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- mm/migrate.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/migrate.c b/mm/migrate.c index de9d94b99ab7..efc19f53b05e 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -262,7 +262,7 @@ static bool remove_migration_pte(struct folio *folio, page_add_anon_rmap(new, vma, pvmw.address, rmap_flags); else - page_add_file_rmap(new, vma, false); + folio_add_file_rmap_pte(folio, new, vma); set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); } if (vma->vm_flags & VM_LOCKED) From patchwork Mon Dec 11 15:56:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487445 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0608DC4167B for ; Mon, 11 Dec 2023 15:57:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AC1256B010E; Mon, 11 Dec 2023 10:57:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A4A2E6B0110; Mon, 11 Dec 2023 10:57:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C1E46B0111; Mon, 11 Dec 2023 10:57:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 78AC06B010E for ; Mon, 11 Dec 2023 10:57:33 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 44B0EA07B2 for ; Mon, 11 Dec 2023 15:57:33 +0000 (UTC) X-FDA: 81554992386.01.D5DDBC0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 771EE20013 for ; Mon, 11 Dec 2023 15:57:31 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=N2+9Zq18; spf=pass (imf13.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310251; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=L2Om8DYO6IhDTMNBMAtiLk55MQgBXdIwfXJDSy3Kty4=; b=jwWQ9VOA3IWjLU7mld04JjzFv3pz3hJxOXoTINN6fAxMV23nHwXWUHpYE59kkHZefn7Poy /LWANkXQvXPmQW/uVH3+zWOqg5KApl0O8g02+NtyMcBuKEhh0t3S1hJsKi6qA+xiHitvIW OANsvi8K31pUnoqJh+tSKzNxh3tEhqw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310251; a=rsa-sha256; cv=none; b=k4gyAg04HHAI5UnVF2CRHeGpn8MBL9XopQUWEzOu0Fj8z0A78wolciPsAeGET0NgvekkpS RWRGuBYFxCIDMvS4yFsQm26pY7bEQK+0PRMOix7DvrgJjacYkYmHJ9YBGNJHMyBgD0CwQB 6hhHGx4QfMDknqvTTkiaNol6AMZ0eQI= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=N2+9Zq18; spf=pass (imf13.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310250; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=L2Om8DYO6IhDTMNBMAtiLk55MQgBXdIwfXJDSy3Kty4=; b=N2+9Zq188uDWOMUQfwFCFm1srABQs5EZoIChGdOc6kC0vEWlVEHlxTw7mjNR1CXEdP+faB tmXXelijASB7rviUqG9UFha86Joy08g/PuxzsPv8g+kWUj7KEAUZZCW7+mel4L+asfP1Mr qQDHzXjnMlSlahw9cein34l5uXRmSrE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-203-TS2p9d5tO1ihFJdn6GaxsQ-1; Mon, 11 Dec 2023 10:57:24 -0500 X-MC-Unique: TS2p9d5tO1ihFJdn6GaxsQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 30A73870829; Mon, 11 Dec 2023 15:57:24 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 900321121306; Mon, 11 Dec 2023 15:57:22 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 11/39] mm/userfaultfd: page_add_file_rmap() -> folio_add_file_rmap_pte() Date: Mon, 11 Dec 2023 16:56:24 +0100 Message-ID: <20231211155652.131054-12-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: 771EE20013 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: ij8fihaykxhbezud7gpqxnrkinqqzp4f X-HE-Tag: 1702310251-684355 X-HE-Meta: U2FsdGVkX1/Q4HJ3azTNMDeqVQ0arsQ+dihHWBKWSXZJot59w+8mst3aJuADTNYJjK9OJpH2oTTwKinzBsZUS7x3GKerNlzQjXTOgwmUL08XmZGoFQDnHWYDMphYBdvvwR5wCUE8BmjYd8dO/fJYxZohk+7zbcjiwvOqp5vXaEj/4PzIq4ZhmEwBmEftJydz0zVxDnTd52hvIoXltvNvJ6mSliWLUq67rh9TybZ56YQSxWK5mPf09HTck6Cs2aHfttOGCC5K6sERG2P4qPpewdmE3+7oR3bosIVJL51vK6FbFC4zMaeS0OUgVLuqf5iAGLKBUL11pR+yaNqbBr+oSwv4mEJhRbqJgkih70U/YjSQ6sbrarMbs+3T5hs5wCNSarRAp8yV/XlIxHZKKYqdBCLW2aY9ewq+6G5FP6t6hOFhETYxYrUY4FI7v8fKk1P1xOgHqIjHlOTdbVFAXOFv2tHPCCpWn3jQ8DKMvHrjrU5dvAJ/vjgImJNEjCi/SFZEZVPCmfW9D6YWomLukW+RYuC0yQbqnNjU3kc5XkFdWVQfnW6n8fxbdOhGzUbJOMzugrgPcmt4YAANEvXAqEqVsfLHmXnUDoDJ2TVwf/fdXc1mImhAqvyEZtAv7NT4Ksrzreu3ABfqtg+2ENufFNycUuuk7hC0fEL+kueSvcebkgCO1Kxdf2rSATYYT7YhHSQa2q9k9w0tD+o9lURgkrajbBozSriUhRWokEvOjPf2tEgSx/LB7I/6E33BFCFGmVgXsjV93BWoJX18+wJguk11+9J5ueKU7t9jE5J3rNkpGuR3ILfscAy2M5ypm82qOWGHr9y20jkF374R1dP8SiiIO8JZN87pCY/Tt37kWKmIgSw8Cf3dOqY6PeqDGb+S4i6VwuBq02zIh9463CcoMgOGPkEuSTAVOZBIPaQOvIDUiElhg+abRXk20V0mmBVAMNfG5ZzZDk831BsIYQjJdMR vUiEaDJs w+ScjA1A9nYMTTrofaNRaG6YSRs0zGae2KfxoqFGobJDnaEEU2ZmHTDL2l+1fqLRXtypKa8xyfvRwxYFBYo5DvZ7GeOog0HYQggpfdgqGluXb3aaEKA5PQCV1STWHXolIA3RDj/PUcW8g+OvpPdGUJPjbEXGmiOezdFbS1Txg1cFYf+jaO3Kllyrsald/haUvnGtgcAGXPN0+ZIzFuoG9FDGPah/KiAikeL1J8u8t9plmfgHvHqvutxuLQw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert mfill_atomic_install_pte(). Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- mm/userfaultfd.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index 9ec814e47e99..330a481a1654 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -114,7 +114,7 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd, /* Usually, cache pages are already added to LRU */ if (newly_allocated) folio_add_lru(folio); - page_add_file_rmap(page, dst_vma, false); + folio_add_file_rmap_pte(folio, page, dst_vma); } else { page_add_new_anon_rmap(page, dst_vma, dst_addr); folio_add_lru_vma(folio, dst_vma); From patchwork Mon Dec 11 15:56:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487444 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9185DC4167B for ; Mon, 11 Dec 2023 15:57:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9FDE86B010C; Mon, 11 Dec 2023 10:57:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 98BD56B010E; Mon, 11 Dec 2023 10:57:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8014D6B010F; Mon, 11 Dec 2023 10:57:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 65E8D6B010C for ; Mon, 11 Dec 2023 10:57:31 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 40E5FA1B81 for ; Mon, 11 Dec 2023 15:57:31 +0000 (UTC) X-FDA: 81554992302.23.E38F058 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 688CE160024 for ; Mon, 11 Dec 2023 15:57:29 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gDLSatI9; spf=pass (imf08.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310249; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DP/TBY2FfVhMMZMh3t2xGvG39Fvp3GsRcAEgonPxhwA=; b=oU9DGtpPlnaOZOpL8Deqx9IdEAJs+Cw1bExLoF8/a329eMLU/0ZsHmXU6eohou4yYWXDds LHcTp42EuGvwbuUIHi2rnU0OPCZCausRftu1d8wwVSsqXi0QimOxV90rVR1bw/iws9RnSb /LLRdxmF0TC63IMGd1Ythn0TmW3yvEI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310249; a=rsa-sha256; cv=none; b=XVpkiBOZ4bumY0bbaWcUB1vXIxQ0offtkx3oppwmSUDTuXvPZuDbd7lKprzl9FGDxAEOu5 M53qEszN4d/m7P2Uv/6P4ji6DQsCi+B/SeBNTNIvVxH+ek5ycim8KJRuc8VH08aPaJb2I7 jkKM7u3wuGCQb27JEmS8GWd5jFeVm5I= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gDLSatI9; spf=pass (imf08.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310248; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DP/TBY2FfVhMMZMh3t2xGvG39Fvp3GsRcAEgonPxhwA=; b=gDLSatI95/KyDsxbQs7ZxnoLX0WitDLnd1MNChZ1hbQc3Z3AUfDi7ytkANh6efOrg1poIX 8rt22IhJd8cfEmcgSrmEuOo07PaZn4Xi9XLpgpw+bmnWs2ucMLj23Q+lFyFQrikDeNgr7X eYFVyNlNI3Z3smGENs5c3GNiR0xv1ec= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-686-nYAYGRlYOBO4fpab-9MmDA-1; Mon, 11 Dec 2023 10:57:27 -0500 X-MC-Unique: nYAYGRlYOBO4fpab-9MmDA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7362810133A0; Mon, 11 Dec 2023 15:57:26 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9447E1121306; Mon, 11 Dec 2023 15:57:24 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 12/39] mm/rmap: remove page_add_file_rmap() Date: Mon, 11 Dec 2023 16:56:25 +0100 Message-ID: <20231211155652.131054-13-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: 688CE160024 X-Rspam-User: X-Stat-Signature: 6c365h9gi5fhraokgigc3u9fcn8dggid X-Rspamd-Server: rspam03 X-HE-Tag: 1702310249-586922 X-HE-Meta: U2FsdGVkX1/7VtiQesPzJRQsrd6vP/KcuEZ2N0WRsHf63/ocxyYCraa0kjm47naU/pe3m1YwCq7L11zeEK93wvB9kODp+siWcBp8hLNl7IkvnaepyPCAbJXnpYm6EHMypCkPgi+tKWPMAF+uU1VHoBwvIizCNfiLrEP9PxuKpbjCQUxlsP14tTgiq+jOL+PXFfbWdltesd00S9jMaJSTf9iPlqQcPIg1myQqLc2yW2zDdr89HYjixboP4OCZwAFkbu6O2AQSK9vCP3WXGfX5hcXLq8NyqBGr6qcJleeisI4tlDDN7/e6fOnsdFv7YROW8q7XMR0E8C9Qxh0o20/a3thgS57r+hdoep3OxQBs9GJemENokVSW1Gp3BoWpqUOcz3p+PEvN7PpwmlmU7AhYNQF9vAB05AIliJ2sM1PU5Hem4vq5o5qxWTKS8NRLzOZ7a/oiG8PGYU8VK5BSXFT7PpRKhk0y/OZ7P39hwWAzINjboa+VR+VUf1FCADMrvT59NKC76p3TXCb3zQl/frAy5t4vUJhZvdlWvcDluDbFSPEp3n9BNN6htV4BBmQxWvdxE1IEGqInB1Uge9+q9SpprtAIUB/xUGmZp1NiqdbLTO2KMjlEX3siJklIyyi91mgAc45is0xUtNKh6rFMIZj7WLLoYrpLnD4kmxdRtXH5/4yIOH0MS9oBKgb1cHGxYg7fDfg0PXTGB1qPTjvvC5UDXfTQhbJQ3E408MLmvgw0GPwGJ+4wnK703c52yrDvYedzitUzYAa9W9SPNXS3Q01iTdNTwJcz5kv8badrGx4qr/UEUXHn0rc0jGO9Me1DhnZHoiUgZgBN7f7HcMsJU92/YFOqRCAylrIxu/WcPcm6M3whUytTL/cAuL7BHvp8Ou7Vvz3aZP68WeCVoctTAE5JmF0Xz+10TduLNsy59HdItW6ZKoGUkDay9jS9tSzuc9HRAq6+N+62MTwGdyy+kGs /LLzGH7x dLi0DSqvI5ga/GTWjeRXxzb0WoF/gT1GzCcQxo2L7Zk+E1GcTSN9b8js3/1ZRIRA2yTnLPr+jRMEODEfYrk29DMtFaPM6vHOx4F1rk9CQ7BG/eRWHcHbg8AFhRO4Un07x4qEqo0STdvlqTEAfgBFCeDqVsnyhJK07mtutTXO5dyTs+XjdufoMfgYLKOhnrCRD9kd85GlD/GajqBgZLbIyJTDQxo246+7rDPxS68yGFvLtHqmMLNPU3rfJ4A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All users are gone, let's remove it. Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand Reviewed-by: Ryan Roberts --- include/linux/rmap.h | 2 -- mm/rmap.c | 21 --------------------- 2 files changed, 23 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 1753900f4aed..7198905dc8be 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -240,8 +240,6 @@ void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address); void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); -void page_add_file_rmap(struct page *, struct vm_area_struct *, - bool compound); void folio_add_file_rmap_ptes(struct folio *, struct page *, int nr_pages, struct vm_area_struct *); #define folio_add_file_rmap_pte(folio, page, vma) \ diff --git a/mm/rmap.c b/mm/rmap.c index 4f30930a1162..2ff2f11275e5 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1467,27 +1467,6 @@ void folio_add_file_rmap_pmd(struct folio *folio, struct page *page, #endif } -/** - * page_add_file_rmap - add pte mapping to a file page - * @page: the page to add the mapping to - * @vma: the vm area in which the mapping is added - * @compound: charge the page as compound or small page - * - * The caller needs to hold the pte lock. - */ -void page_add_file_rmap(struct page *page, struct vm_area_struct *vma, - bool compound) -{ - struct folio *folio = page_folio(page); - - VM_WARN_ON_ONCE_PAGE(compound && !PageTransHuge(page), page); - - if (likely(!compound)) - folio_add_file_rmap_pte(folio, page, vma); - else - folio_add_file_rmap_pmd(folio, page, vma); -} - /** * page_remove_rmap - take down pte mapping from a page * @page: page to remove mapping from From patchwork Mon Dec 11 15:56:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487446 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DD46C4167B for ; Mon, 11 Dec 2023 15:57:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F3376B0110; Mon, 11 Dec 2023 10:57:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 47C906B0112; Mon, 11 Dec 2023 10:57:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 31D4D6B0113; Mon, 11 Dec 2023 10:57:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 1EEF66B0110 for ; Mon, 11 Dec 2023 10:57:35 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id F10F116072C for ; Mon, 11 Dec 2023 15:57:34 +0000 (UTC) X-FDA: 81554992428.15.5A84FAA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf15.hostedemail.com (Postfix) with ESMTP id 386D2A0008 for ; Mon, 11 Dec 2023 15:57:32 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BZTQcmwn; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf15.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310253; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FoY+MxvyMEZDCAVG6BwT9WrYolbRNCvL6DZB18ZE48E=; b=F288ejRE+ih5ZE4OOaPrirvpPGnXaIX4Y0L1ECQZ6k2U4CDMxanGxORPAzpDkin4rHVoRG N5c4vSZc154w7GEd4ZqFTcX+FWm/lhplDHwtcqohTXq234I6Mr3yJ5imQUYmsCWFl6MqJg Ae9qwM2mJOYdH8A8Bctx2pqyCnaYXgo= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BZTQcmwn; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf15.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310253; a=rsa-sha256; cv=none; b=RcC504P/lL2OkA7gJEjbE3h4OTGaycwrcoiE8E1jIcZbLSPJnOxnzdlDT55L95w+7RA2QE XBovFw+1fvrr+Pun9aB500lZuNmSm6skMeETH7Gb1QUC9lzPv0eKOLPERSdlFv9k38bwKX jlT9oCA8oATrIadWjsU1BuwKc7RrEUc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310252; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FoY+MxvyMEZDCAVG6BwT9WrYolbRNCvL6DZB18ZE48E=; b=BZTQcmwnHEE3NhoPft/JbJBJnjFaHflB76whuAcMKF5XldlXF2PPO/rSmTeIbwfmWyfugB xeKj/PpF6hyShvQTHVJmP7K6HbMdPVfRCCe9cxRp/l0NBwjZLxZHnvA05crFtkcIvDBuay Xjyt8702tkGC0CMc5oPilUKLEdMttpI= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-632-2BLn_LvuPamqZYZJzhfDrg-1; Mon, 11 Dec 2023 10:57:29 -0500 X-MC-Unique: 2BLn_LvuPamqZYZJzhfDrg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 1EEF9870822; Mon, 11 Dec 2023 15:57:29 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id D51861121306; Mon, 11 Dec 2023 15:57:26 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 13/39] mm/rmap: factor out adding folio mappings into __folio_add_rmap() Date: Mon, 11 Dec 2023 16:56:26 +0100 Message-ID: <20231211155652.131054-14-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: 386D2A0008 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 8tdhyaui86q5147pke599iya7icfgj7r X-HE-Tag: 1702310252-662858 X-HE-Meta: U2FsdGVkX1/QZ9lNgv5AlK2WjkX8FsNtgxYoH1fjMiX6AG6X670WuOWQr0AZRwajsBsX67Ip89Rctq9y7CVpe95jbzb32PX6T3IqrXWaauhhetMj4R8PGhRzF4T/OXt8CwZVBDRGq8xoRV4VyhR4Dn/VJrbvdHAJ0mTCjhR0lQkYFe5qE9QEbZlw0XxbL/LFKPvpuS8BwQovwgQ/l6kEkbMcC1rvUuN1ZsokKPGB0XHpvGzIbbCZ4sfitLNZKhdrIIcVtwMs5AnfV/xx81bLhXunrZxeQwUimfQodJqHUuJqzyZi+s+0WJqICge6Wt0NT4gsflLlZ4ORVXqvCltSOrowuQEbZwSkit3QXdIvXK6SB5jYWgNTCCZAsDqq1g2JMXS6qbqEiiUb3lMLpvgolqC76dwPs7hIiAn9CG7IX+ponJkPvt2/FQkAoH/3Aon8kDIzpOwQ+edxd9n8zYrZqXemG+HGU3YXHJ80WhDBtujR5haMDnh4JyQVGKCc2PCBpkbalbGI4mQaRpxszSNZRg9VtdQNULTtc0GNSW1uwFSC585kPRg0NiQ1VtHhTM6nJKO7CN8VIG4GvhBAwUkjpoaaV9ZPaig+LcjHfvVPRgC976iFwHUY50r0LLgzYt6Zhq0uzoffiQEm9/obfscDYJtVfzOezV/OGLcP68xdOcF5/Gfw1PuLXxTnM/8NcKloEJ3xeItzEjDkpxu3mQ9aTSV9ttz+M8/KRDxOPE/3okzyAwhVcZpzpGe6UFDknf0qm00m+ht9MAL59ebHvcPC78ZHkEFlLhnyZh0NJOK6Jb6lvHyxr+PRjIQ1tU7ivT1ZEcQGwb5/ZMyErMLJtBZ385S999jYJTKnO1T9rvQ0h9YBNA/fNS4cv7XAqvDeanSloCMl6GeDFppKZnTQtfpgcUHeToYTWHGcdQPfR5lsemtwWsHVXDFciWgrZufscDMuIAk9QQVXVTR8KWTo1lQ Pii/1tVx YpUuFoNLNWY6d/rwFMSxnYpODDJ6DJT8A3hp6x9fnd2NfkPDMjo7wb0un9kxSv19adGNAIdIrw4ywIvo3TOlRUhzi+XGAR8UHm5b7+ZxFFBwFe2TAaFNyEpmY/iGm1qgWCaHx5vjf3EOkSe7Bh/QcZ/wNB1yDG/YlU1mSTktxEHqKTKH6CUXBe9Pp/bREu4hrDMJCaMA7niSEd+JzGxFaLXRAWZ0jYAIvXoGhywCvoBDEi3oeo+yjLkUSCw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's factor it out to prepare for reuse as we convert page_add_anon_rmap() to folio_add_anon_rmap_[pte|ptes|pmd](). Make the compiler always special-case on the granularity by using __always_inline. Reviewed-by: Yin Fengwei Signed-off-by: David Hildenbrand --- mm/rmap.c | 81 ++++++++++++++++++++++++++++++------------------------- 1 file changed, 45 insertions(+), 36 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index 2ff2f11275e5..c5761986a411 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1157,6 +1157,49 @@ int folio_total_mapcount(struct folio *folio) return mapcount; } +static __always_inline unsigned int __folio_add_rmap(struct folio *folio, + struct page *page, int nr_pages, enum rmap_mode mode, + unsigned int *nr_pmdmapped) +{ + atomic_t *mapped = &folio->_nr_pages_mapped; + int first, nr = 0; + + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); + + /* Is page being mapped by PTE? Is this its first map to be added? */ + switch (mode) { + case RMAP_MODE_PTE: + do { + first = atomic_inc_and_test(&page->_mapcount); + if (first && folio_test_large(folio)) { + first = atomic_inc_return_relaxed(mapped); + first = (first < COMPOUND_MAPPED); + } + + if (first) + nr++; + } while (page++, --nr_pages > 0); + break; + case RMAP_MODE_PMD: + first = atomic_inc_and_test(&folio->_entire_mapcount); + if (first) { + nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); + if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { + *nr_pmdmapped = folio_nr_pages(folio); + nr = *nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); + /* Raced ahead of a remove and another add? */ + if (unlikely(nr < 0)) + nr = 0; + } else { + /* Raced ahead of a remove of COMPOUND_MAPPED */ + nr = 0; + } + } + break; + } + return nr; +} + /** * folio_move_anon_rmap - move a folio to our anon_vma * @folio: The folio to move to our anon_vma @@ -1380,45 +1423,11 @@ static __always_inline void __folio_add_file_rmap(struct folio *folio, struct page *page, int nr_pages, struct vm_area_struct *vma, enum rmap_mode mode) { - atomic_t *mapped = &folio->_nr_pages_mapped; - unsigned int nr_pmdmapped = 0, first; - int nr = 0; + unsigned int nr, nr_pmdmapped = 0; VM_WARN_ON_FOLIO(folio_test_anon(folio), folio); - __folio_rmap_sanity_checks(folio, page, nr_pages, mode); - - /* Is page being mapped by PTE? Is this its first map to be added? */ - switch (mode) { - case RMAP_MODE_PTE: - do { - first = atomic_inc_and_test(&page->_mapcount); - if (first && folio_test_large(folio)) { - first = atomic_inc_return_relaxed(mapped); - first = (first < COMPOUND_MAPPED); - } - - if (first) - nr++; - } while (page++, --nr_pages > 0); - break; - case RMAP_MODE_PMD: - first = atomic_inc_and_test(&folio->_entire_mapcount); - if (first) { - nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); - if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { - nr_pmdmapped = folio_nr_pages(folio); - nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); - /* Raced ahead of a remove and another add? */ - if (unlikely(nr < 0)) - nr = 0; - } else { - /* Raced ahead of a remove of COMPOUND_MAPPED */ - nr = 0; - } - } - break; - } + nr = __folio_add_rmap(folio, page, nr_pages, mode, &nr_pmdmapped); if (nr_pmdmapped) __lruvec_stat_mod_folio(folio, folio_test_swapbacked(folio) ? NR_SHMEM_PMDMAPPED : NR_FILE_PMDMAPPED, nr_pmdmapped); From patchwork Mon Dec 11 15:56:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0DF2C10F13 for ; Mon, 11 Dec 2023 15:57:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B84BA6B0112; Mon, 11 Dec 2023 10:57:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B37646B0114; Mon, 11 Dec 2023 10:57:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 962696B0115; Mon, 11 Dec 2023 10:57:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8277A6B0112 for ; Mon, 11 Dec 2023 10:57:38 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 4B509A06CB for ; Mon, 11 Dec 2023 15:57:38 +0000 (UTC) X-FDA: 81554992596.14.76FC29C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf16.hostedemail.com (Postfix) with ESMTP id 9684A180011 for ; Mon, 11 Dec 2023 15:57:36 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="MkYo/Z9E"; spf=pass (imf16.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310256; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FvIn4PK1R1vqfE/cFFREYzckJVopW/NXl4TYpNX8kUk=; b=eXo7juHofT0/YIgjB+X42vAJ88Vy2wm97TpjoU37OW40Zf5VqROTog9cClRlQWilJzzLTx HzMohczUImgPYYdpcXsDfh/9KUVuj/baMIWgn9q//8MKXX9mKcz6i8hhQSqvFCxImTFb40 yzdfOis0WxND3QQybze+N0dvsT/X1yM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310256; a=rsa-sha256; cv=none; b=aV+TsyNNeMRQiVYkHW4Pd6thDf1ThSu5F5qfQnS90Zat1eBDY7YzmQuPZR56t1H3uagxoc zvZD8qDqrEczRrFnra/alK5FZFu6Wn3etu9V00Jfc4EJFund8BYGDH/ITUioer27BekqHI 4xK/AC+Gah652r1QxiG2BlO6QHM1iKs= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="MkYo/Z9E"; spf=pass (imf16.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310255; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FvIn4PK1R1vqfE/cFFREYzckJVopW/NXl4TYpNX8kUk=; b=MkYo/Z9E6GScHnS2rT58BD8uNQTw4AhokrXrRuUriv5AQ1KS3HAgRCzxzsX3MtTjmEhWuF hQtGXZqBgPre+a5hXPDdZ38Iu+p8SEeO2RoiLf9/o387br3tSGbWIk44WIWz2A/OD3C1Ye giK6FqmUb1PL0S52PlCDvRqztgFzTTc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-694-bp0xtxh7M-6waMAcw-G2oA-1; Mon, 11 Dec 2023 10:57:32 -0500 X-MC-Unique: bp0xtxh7M-6waMAcw-G2oA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id AA129870820; Mon, 11 Dec 2023 15:57:31 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 583EB1121306; Mon, 11 Dec 2023 15:57:29 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 14/39] mm/rmap: introduce folio_add_anon_rmap_[pte|ptes|pmd]() Date: Mon, 11 Dec 2023 16:56:27 +0100 Message-ID: <20231211155652.131054-15-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Stat-Signature: ufxgerkp37jznzwooixm3unaa1uco37r X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 9684A180011 X-Rspam-User: X-HE-Tag: 1702310256-341376 X-HE-Meta: U2FsdGVkX1/tcZvd47G85SHNGNBkSlYI53CJbN4p9WncG9Jg7UQ0C70Xu0L0vvksWlZIKfMIEK/XQUx8XLBaF1Aj1nXT29or0TiNftwCgW90r6Gk8Csb2RiyEhf2PU9mnXTMdsagLWtb4yo4gWO4ezUm+lJmy6ZAVD5P1RJcGGGANwTd+0UtdeGbNLZbism1JIPQgX1qP7W5rZCf5GHjq+EQCk4SNfGFWYDaEjWOya90dzb9M5mk+ZRLFsNKEEEIRhDWLKBbqPq0LjzEKku56bBAfn5uTTTQoF3JrRYFO+3aXfzHUstXhOEmaMK/qBICsfOyZSr1Rvb4gh9QXMiVrJy2YiSmvXGOBzktODt7I+5tolTGk6KsBfagYlLIIkRquwhSP4vTooq4DgWJEc+cZ59usOudehfVfBOLZWNv4JhGS9LyHNLeUFoa6UNaEzi3ZoBkz9FwhYQxtdRiX7yiwQPpBX1sb/pFD51efnHPjeIOy0tjtrPUamiG4S3XZ35XkyMK+ggEhwPMrZa2TLJvtze0l1+bupVv0Z1FajD+rLltkKYH4AKkfZD+wenz+mLcHhoLCb46/ogoUh5trDfN2BBWQPt3QovT7J/ppQQ59xY6AMzBcgS6OAiqHl2t94eUKRt/rT/IMUSUtwO+BenENbXdd3cPq/Zyj6/CjgJ9KJ0F+smvuIIyxaRwmpy/12yr+UHwPat0fttcA1TJTegyOIGQ7lfsMjTTxgPxCOfPaN9P4RDUPHGxVCTAjstOy18C/UZzcBWLZkSFjgyavOmrTNnfFZtBCCd45LsmpEWXOqlk/vX3vJFqQFQyVEJlVkbkwNlqJiVGHrkrArKG0joR5KTfeCxHFxBZaz44De4GJWREvPO+zV7ynWEb/oPMzUfHhKu9Uv6Cluwt/IHhYNGnAMNW3vqrfmt0vIut/uY9CHql1GUoFcItTNAvbxfLOCtK3tfNVFD4XaUZUTMmBSa wdxFX4QW h2DMRLE8iDEDAKiOnTfW5fS/blpUc+iIP5cdWzMrB/gvXFJSCpavpcU+0obIOrgIGTYED0tA3XAEvU4fQcQLKfqBMfwtx3Qbn9qovZThq9igz2aI/Mkdm1ZMHdczNwt3zMk/LC1wI/kIHfoXRkaiKVp2UEz2fVPt0leLzV+IGeU47R4JHKIQ0fVLELWGkldtaJ8H9R1n/WIDc8YcXjY+DWMPcmMxpOX5CS2uT X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's mimic what we did with folio_add_file_rmap_*() so we can similarly replace page_add_anon_rmap() next. Make the compiler always special-case on the granularity by using __always_inline. Note that the new functions ignore the RMAP_COMPOUND flag, which we will remove as soon as page_add_anon_rmap() is gone. Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei --- include/linux/rmap.h | 6 +++ mm/rmap.c | 118 ++++++++++++++++++++++++++++++------------- 2 files changed, 88 insertions(+), 36 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 7198905dc8be..3b5357cb1c09 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -234,6 +234,12 @@ static inline void __folio_rmap_sanity_checks(struct folio *folio, * rmap interfaces called when adding or removing pte of page */ void folio_move_anon_rmap(struct folio *, struct vm_area_struct *); +void folio_add_anon_rmap_ptes(struct folio *, struct page *, int nr_pages, + struct vm_area_struct *, unsigned long address, rmap_t flags); +#define folio_add_anon_rmap_pte(folio, page, vma, address, flags) \ + folio_add_anon_rmap_ptes(folio, page, 1, vma, address, flags) +void folio_add_anon_rmap_pmd(struct folio *, struct page *, + struct vm_area_struct *, unsigned long address, rmap_t flags); void page_add_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address, rmap_t flags); void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, diff --git a/mm/rmap.c b/mm/rmap.c index c5761986a411..7787499fa2ad 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1300,38 +1300,20 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, unsigned long address, rmap_t flags) { struct folio *folio = page_folio(page); - atomic_t *mapped = &folio->_nr_pages_mapped; - int nr = 0, nr_pmdmapped = 0; - bool compound = flags & RMAP_COMPOUND; - bool first; - /* Is page being mapped by PTE? Is this its first map to be added? */ - if (likely(!compound)) { - first = atomic_inc_and_test(&page->_mapcount); - nr = first; - if (first && folio_test_large(folio)) { - nr = atomic_inc_return_relaxed(mapped); - nr = (nr < COMPOUND_MAPPED); - } - } else if (folio_test_pmd_mappable(folio)) { - /* That test is redundant: it's for safety or to optimize out */ + if (likely(!(flags & RMAP_COMPOUND))) + folio_add_anon_rmap_pte(folio, page, vma, address, flags); + else + folio_add_anon_rmap_pmd(folio, page, vma, address, flags); +} - first = atomic_inc_and_test(&folio->_entire_mapcount); - if (first) { - nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); - if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { - nr_pmdmapped = folio_nr_pages(folio); - nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); - /* Raced ahead of a remove and another add? */ - if (unlikely(nr < 0)) - nr = 0; - } else { - /* Raced ahead of a remove of COMPOUND_MAPPED */ - nr = 0; - } - } - } +static __always_inline void __folio_add_anon_rmap(struct folio *folio, + struct page *page, int nr_pages, struct vm_area_struct *vma, + unsigned long address, rmap_t flags, enum rmap_mode mode) +{ + unsigned int i, nr, nr_pmdmapped = 0; + nr = __folio_add_rmap(folio, page, nr_pages, mode, &nr_pmdmapped); if (nr_pmdmapped) __lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr_pmdmapped); if (nr) @@ -1345,18 +1327,34 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, * folio->index right when not given the address of the head * page. */ - VM_WARN_ON_FOLIO(folio_test_large(folio) && !compound, folio); + VM_WARN_ON_FOLIO(folio_test_large(folio) && + mode != RMAP_MODE_PMD, folio); __folio_set_anon(folio, vma, address, !!(flags & RMAP_EXCLUSIVE)); } else if (likely(!folio_test_ksm(folio))) { __page_check_anon_rmap(folio, page, vma, address); } - if (flags & RMAP_EXCLUSIVE) - SetPageAnonExclusive(page); - /* While PTE-mapping a THP we have a PMD and a PTE mapping. */ - VM_WARN_ON_FOLIO((atomic_read(&page->_mapcount) > 0 || - (folio_test_large(folio) && folio_entire_mapcount(folio) > 1)) && - PageAnonExclusive(page), folio); + + if (flags & RMAP_EXCLUSIVE) { + switch (mode) { + case RMAP_MODE_PTE: + for (i = 0; i < nr_pages; i++) + SetPageAnonExclusive(page + i); + break; + case RMAP_MODE_PMD: + SetPageAnonExclusive(page); + break; + } + } + for (i = 0; i < nr_pages; i++) { + struct page *cur_page = page + i; + + /* While PTE-mapping a THP we have a PMD and a PTE mapping. */ + VM_WARN_ON_FOLIO((atomic_read(&cur_page->_mapcount) > 0 || + (folio_test_large(folio) && + folio_entire_mapcount(folio) > 1)) && + PageAnonExclusive(cur_page), folio); + } /* * For large folio, only mlock it if it's fully mapped to VMA. It's @@ -1368,6 +1366,54 @@ void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, mlock_vma_folio(folio, vma); } +/** + * folio_add_anon_rmap_ptes - add PTE mappings to a page range of an anon folio + * @folio: The folio to add the mappings to + * @page: The first page to add + * @nr_pages: The number of pages which will be mapped + * @vma: The vm area in which the mappings are added + * @address: The user virtual address of the first page to map + * @flags: The rmap flags + * + * The page range of folio is defined by [first_page, first_page + nr_pages) + * + * The caller needs to hold the page table lock, and the page must be locked in + * the anon_vma case: to serialize mapping,index checking after setting, + * and to ensure that an anon folio is not being upgraded racily to a KSM folio + * (but KSM folios are never downgraded). + */ +void folio_add_anon_rmap_ptes(struct folio *folio, struct page *page, + int nr_pages, struct vm_area_struct *vma, unsigned long address, + rmap_t flags) +{ + __folio_add_anon_rmap(folio, page, nr_pages, vma, address, flags, + RMAP_MODE_PTE); +} + +/** + * folio_add_anon_rmap_pmd - add a PMD mapping to a page range of an anon folio + * @folio: The folio to add the mapping to + * @page: The first page to add + * @vma: The vm area in which the mapping is added + * @address: The user virtual address of the first page to map + * @flags: The rmap flags + * + * The page range of folio is defined by [first_page, first_page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock, and the page must be locked in + * the anon_vma case: to serialize mapping,index checking after setting. + */ +void folio_add_anon_rmap_pmd(struct folio *folio, struct page *page, + struct vm_area_struct *vma, unsigned long address, rmap_t flags) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + __folio_add_anon_rmap(folio, page, HPAGE_PMD_NR, vma, address, flags, + RMAP_MODE_PMD); +#else + WARN_ON_ONCE(true); +#endif +} + /** * folio_add_new_anon_rmap - Add mapping to a new anonymous folio. * @folio: The folio to add the mapping to. From patchwork Mon Dec 11 15:56:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487448 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EAD05C4167B for ; Mon, 11 Dec 2023 15:57:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7DB836B0114; Mon, 11 Dec 2023 10:57:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 789A16B0115; Mon, 11 Dec 2023 10:57:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 605F66B0116; Mon, 11 Dec 2023 10:57:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 495466B0114 for ; Mon, 11 Dec 2023 10:57:41 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 276781A0674 for ; Mon, 11 Dec 2023 15:57:41 +0000 (UTC) X-FDA: 81554992722.12.3E94659 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf16.hostedemail.com (Postfix) with ESMTP id 53E66180023 for ; Mon, 11 Dec 2023 15:57:39 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JU+jXHi5; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf16.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310259; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gWnH4eIM4y9lGkdEQ6xWZD6LXOR6bYxTBmelfOqDsXI=; b=aoPM0KDIic3cJPai/TYd2AignSgDGGVUVoo15LXTMwwTkZvGpLSzTXp5JoFTVLoq5QJ5OU PY2GwDteIhL5qgQjVjB8jbRgm38d9AuemdTQ/Bmk6NBuneQV1fTPSvcT7MbR4iAt6rzBiC d2kIh6EH/oTOFkCjsPtVMZWkM5X3FhQ= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JU+jXHi5; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf16.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310259; a=rsa-sha256; cv=none; b=gt8w9aDN86FV6vQCSjAbF75XN8w/+XGN/qxkC7LjOdYmqZ7xpxvYYSiOLZha9EsHGiMjNc 7CMYaNluOFmb7k4+ngqL4ck6CJ9d1NPQNuFp4L9BbFaoLigHfUnprneJxCOcSdaHKM4n/v 6uA1uGOuCc2wuT+yS5oNBHqaE57NcdU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310258; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gWnH4eIM4y9lGkdEQ6xWZD6LXOR6bYxTBmelfOqDsXI=; b=JU+jXHi53Tz/FWXnxM4ejB5AzKvOp2hzWMbwsggpnvOwEn3S1VeDOaCIr24QRCkBnhAolz FDjEY94RfmiaS0exQoEBu9Ly6/6lf7yAkyOk4UgbRwEzVovgRSyG1LsyYferaPszjH93CD xZOqZhhi9FqnBXvVkxGrlJPkJa4ahuc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-255-Z3gSjZ9jPqqQxyKFZIs1PA-1; Mon, 11 Dec 2023 10:57:35 -0500 X-MC-Unique: Z3gSjZ9jPqqQxyKFZIs1PA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 742D1185A788; Mon, 11 Dec 2023 15:57:34 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id EF3641121306; Mon, 11 Dec 2023 15:57:31 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 15/39] mm/huge_memory: batch rmap operations in __split_huge_pmd_locked() Date: Mon, 11 Dec 2023 16:56:28 +0100 Message-ID: <20231211155652.131054-16-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspam-User: X-Stat-Signature: ee6491nqn3hfpa5wg83kj1ztichuxz77 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 53E66180023 X-HE-Tag: 1702310259-618705 X-HE-Meta: U2FsdGVkX1/p6R1ihPZzYIYGpIlp7yQsZSSRZAznjX7Bld6YhD3tAHsf8X+azcUCGxW6Z5DpNukKwHjDZ3H35g8Xuj7ZPLchuSDhgFmkt1uo9Edk8wH3yjWmPQ6iz0UGcnkj5OD7GWaQoPEFpZSfCrktwaDdNJ8/t3UsP7jp8lKEwGENNNgWm7mDn/ZvXQIVjF6Ytmk+jOhkaiQx8GNVf+MQtDM3BQ5tRcmmgbZpstYDEUL95lxaP0Def13MozW2KX9+PM4FKRb4p1mH8/KPKrwszseMHVVTBdzzDZ3gQD/xVSaumJYiyN28hBfdzFC08cLPzsnR1TH5fS5RtkEejIm3xhNjoq3yQI/4aE+1Klls2+PBxVfglWG6pCo0GBO3qleB9rp5PR7pGvWT0UNsRkdh9RC8h4ZdnQUHzvKO6uoMXqhDAkI+m+EoR4zrAxVUmD3wuhbKpwagz9ZzHoDYvOzRkhSFIxezXgT7JJx7jCaq0Y/mLHrhkgmn9HbPes83+uAMo4C9m2W7S2itOVKw7MraV/iMOzeerR7zOylWkvRK8pD717c0r17jFjuyMy5XlIaUEix/kX1ngGT7CliEYxRw6+WNc/NfUjRrO0mLJbtymsekljgSBxIEBr0XX++QJrBeCuvfCvjuHlf4cYAZOMWRiqctDoSA5P63ZeDa6a6FXN8v/OVKW0U8gHcNFY3d2oQlUJwZ8uuhExKytRBh2ZdHEnjwS0nevqk0p+azXnRduk27RRzBXY4l3gSdKin8acUDcqVHXMI7jnMk8+R9Z/J76aeLJAcapHOFLpEKW+KTbq+f3Q0riZQDyVdPeDTE84+YaB5CEoy7b7grspAkwDICHHnthu5alPX3JJ1QlhwbQ7s14eKYiRc6lZglQNT84cxD13BlAP+4qFAPCpqR3Bb9RDs1+GQwgfNuXVLhpjGI6vN4R+3WMEyZTvzzl+BBNZ36uMlLFa5Y5e5Ej5D +TxxBT11 maMQNErScISNHT/SnsvyoPDTr6HvFvJdLKyYCO8ubed6MqbZyQkqw1OTLl2hkPYlRILYtz3mOcI6BJG1R04J9ck/ukne7bxC5TZCWdD0RoaEs+CmT+/+2yB8Nr4rtfUZqYiQudTIUgo1952+q3ffbqEpsR2QZwlhbZkPjiFIPhJnmQIeZMrNweXnOfqGd9RHdRj3GWLPSKkr+fnz9sG6Frrz6xA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's use folio_add_anon_rmap_ptes(), batching the rmap operations. While at it, use more folio operations (but only in the code branch we're touching), use VM_WARN_ON_FOLIO(), and pass RMAP_EXCLUSIVE instead of manually setting PageAnonExclusive. We should never see non-anon pages on that branch: otherwise, the existing page_add_anon_rmap() call would have been flawed already. Signed-off-by: David Hildenbrand Reviewed-by: Yin Fengwei Reviewed-by: Yin Fengwei --- mm/huge_memory.c | 23 +++++++++++++++-------- 1 file changed, 15 insertions(+), 8 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 1f5634b2f374..82ad68fe0d12 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2398,6 +2398,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, unsigned long haddr, bool freeze) { struct mm_struct *mm = vma->vm_mm; + struct folio *folio; struct page *page; pgtable_t pgtable; pmd_t old_pmd, _pmd; @@ -2493,16 +2494,18 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, uffd_wp = pmd_swp_uffd_wp(old_pmd); } else { page = pmd_page(old_pmd); + folio = page_folio(page); if (pmd_dirty(old_pmd)) { dirty = true; - SetPageDirty(page); + folio_set_dirty(folio); } write = pmd_write(old_pmd); young = pmd_young(old_pmd); soft_dirty = pmd_soft_dirty(old_pmd); uffd_wp = pmd_uffd_wp(old_pmd); - VM_BUG_ON_PAGE(!page_count(page), page); + VM_WARN_ON_FOLIO(!folio_ref_count(folio), folio); + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); /* * Without "freeze", we'll simply split the PMD, propagating the @@ -2519,11 +2522,18 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, * * See page_try_share_anon_rmap(): invalidate PMD first. */ - anon_exclusive = PageAnon(page) && PageAnonExclusive(page); + anon_exclusive = PageAnonExclusive(page); if (freeze && anon_exclusive && page_try_share_anon_rmap(page)) freeze = false; - if (!freeze) - page_ref_add(page, HPAGE_PMD_NR - 1); + if (!freeze) { + rmap_t rmap_flags = RMAP_NONE; + + folio_ref_add(folio, HPAGE_PMD_NR - 1); + if (anon_exclusive) + rmap_flags |= RMAP_EXCLUSIVE; + folio_add_anon_rmap_ptes(folio, page, HPAGE_PMD_NR, + vma, haddr, rmap_flags); + } } /* @@ -2566,8 +2576,6 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = mk_pte(page + i, READ_ONCE(vma->vm_page_prot)); if (write) entry = pte_mkwrite(entry, vma); - if (anon_exclusive) - SetPageAnonExclusive(page + i); if (!young) entry = pte_mkold(entry); /* NOTE: this may set soft-dirty too on some archs */ @@ -2577,7 +2585,6 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = pte_mksoft_dirty(entry); if (uffd_wp) entry = pte_mkuffd_wp(entry); - page_add_anon_rmap(page + i, vma, addr, RMAP_NONE); } VM_BUG_ON(!pte_none(ptep_get(pte))); set_pte_at(mm, addr, pte, entry); From patchwork Mon Dec 11 15:56:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9910EC4167B for ; Mon, 11 Dec 2023 15:57:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8ECCE6B0115; Mon, 11 Dec 2023 10:57:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 89B066B0116; Mon, 11 Dec 2023 10:57:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 717E96B0117; Mon, 11 Dec 2023 10:57:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 5A0A56B0115 for ; Mon, 11 Dec 2023 10:57:45 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2FAB5807AA for ; Mon, 11 Dec 2023 15:57:45 +0000 (UTC) X-FDA: 81554992890.05.9122CEA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 6A69240009 for ; Mon, 11 Dec 2023 15:57:43 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bZcFdd6B; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310263; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UzjxFSmJorC6pNmLrufiiZpWu6JbL+HXYJ1oUv8JfJw=; b=bM9s1iMct6EIWYDKYVYy08CmkKhYL+dvrJpbKKeF0YPDYnQ+XA9s72rNtESHLLFD9byLuJ xVMzFi9Qg2EYcXsh94gx7SafSX+rJ11+1Mxq82rzISRIO/8jKiljO5z/ut/0QHvpugOZsC Xt0eQRWSb8Yfkj0uCLI81/5rF+lDDF4= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bZcFdd6B; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310263; a=rsa-sha256; cv=none; b=UrdTwG+hPXxtQOSx3nEMkGu06wBM8CygeWpBXcQQPzkQaEJsduscIXEkIQ6lqQmq5QmfK7 WDD2x1lbh4LEexxRSrjpGebqWn2YZqyCGQAtD2702aPI+7VoA73vUjLP9NRFHdrcYnTX1o pkq558qroej2dxIDDwziIZ6z8kDMQlE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310262; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UzjxFSmJorC6pNmLrufiiZpWu6JbL+HXYJ1oUv8JfJw=; b=bZcFdd6BVUyaYYfcHuvhyr93xFNCZ3u6Bn1bLXofhTrOJbZ6Jc/OdBliYDA9AhBH+F4j/E RxeNsqIA2awpvEFMRmHze1OA/oO7J96juLMCGjej1phmIEz/8BunrBuuR5oIhZQa9PBUlp MLqvjd5Ossifz78SxnyuKtaWnxdldu0= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-180-2Y4Cj1kpO9CR1D_HN3qvZQ-1; Mon, 11 Dec 2023 10:57:37 -0500 X-MC-Unique: 2Y4Cj1kpO9CR1D_HN3qvZQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id BB36238425B3; Mon, 11 Dec 2023 15:57:36 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id D12261121306; Mon, 11 Dec 2023 15:57:34 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 16/39] mm/huge_memory: page_add_anon_rmap() -> folio_add_anon_rmap_pmd() Date: Mon, 11 Dec 2023 16:56:29 +0100 Message-ID: <20231211155652.131054-17-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: 6A69240009 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: ert3z16woa4jni6tks51s4eo9kuemu39 X-HE-Tag: 1702310263-639007 X-HE-Meta: U2FsdGVkX1/wKtlYPvjOjNGB0YEljPM+zGPqWwk/uchAqr/VK4Wj7VNyHpAxvos0QaXDRrCIHvAootIq1/iEvpIA1Jh/6K4BOTIyGMFRP/bQ+4kQtF61abZi0v9GdPlJzRBrjor3xO8TygGyaDR7NsqWfkiws61z/KAq5MY5l/pOGU/UGjH1hbSDmdnvBDRDiHUy2Zu5nL7TXQgUScrEK6WYoaOg0V7l7uF4LZaycAgpLcFnI0aiqWpUNh9pvFPoLRviSyHS1m7jX3muXR0+43KtSkYGO2kU9bGK8YtjkMc/c7nIcwUUuX8Bv2+Sbn72zL/oaG6iEKzu10+Vf0HXXa/eXDXI5bSKWRJDiab0FEP2ICDYSUCjVHPsQ3Eo0phUw7hOi7PmXlJ7gYfG6mr5SvfOwbns1Eu2y6X3g2Bt+aJsH9eWPHrAByiJGe89yXtTLWozKn5ePA8s+575keTaignk7Qhgs9A37YqkrvX0v0nA6OqvVyzC9YqXd16P7YN8w1285keWjbdS4qpesVoItZMejdi0Y7WwPbZk25TvuFlqOGas1QlCoY8L3vk22GygwIWrlOaf6ySvIi0KPMkvLt2XQ5XnhHauj3RNRFaS/MGqQTHxYe2fKdaMaH76M0NCT5HpvNQli3pt33lptQSjKgZd1ZiNz2mUXgyQIU2CjgLcV3KCOBNH8uytKQ1AREOPQcCWxt/1tmbUHWL58Enl2Vp78xoCu3gS9Ss+nwexuYKdaS+TEQdHNGnje+kGY6iGIqXqP6D+VzfUNzUBMrcNgFmOjZUCYk/0UkpxP1PKgYYG7K68acub9nh5LnRFt8nf9PAqTucfhsMe4bjrMa+u61u95JRfhvfZax64HqIz+4ZC2+Qu/QF0W5Gi5h5Lhmy9dLaM9H408E2VWMYK7nNCnaumrFut8+TiU2eJsmeNFSap5bT2iUBrV01YefKJRnaeyFkxLsMPACutqkSuUlf RfpRkSrc YVY8eozzVZWLhw3xfy1fJvIMp2P+//XBgI8sv1AUlXsqlcnZDBS2qcaDModM8E+Do/uU0YNFtH6XNvjGhJo927ShWgeo1OedTZ5A+1gvdqXm4BdXX2BNK6v7Ik425f9APYxbtdsKCN8fD7PAu69JuK9t7/xyXIfKx5attFL4nLtcs08y4DNbaLrfnsd9lAAcnp4Z+56FuTA9imHuc3cHsh7oZcQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert remove_migration_pmd(). No need to set RMAP_COMPOUND, that we will remove soon. Signed-off-by: David Hildenbrand --- mm/huge_memory.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 82ad68fe0d12..b03374d1bb94 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3611,12 +3611,12 @@ void remove_migration_pmd(struct page_vma_mapped_walk *pvmw, struct page *new) pmde = pmd_mkdirty(pmde); if (folio_test_anon(folio)) { - rmap_t rmap_flags = RMAP_COMPOUND; + rmap_t rmap_flags = RMAP_NONE; if (!is_readable_migration_entry(entry)) rmap_flags |= RMAP_EXCLUSIVE; - page_add_anon_rmap(new, vma, haddr, rmap_flags); + folio_add_anon_rmap_pmd(folio, new, vma, haddr, rmap_flags); } else { folio_add_file_rmap_pmd(folio, new, vma); } From patchwork Mon Dec 11 15:56:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487457 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90BD4C4167B for ; Mon, 11 Dec 2023 15:57:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4CF996B0117; Mon, 11 Dec 2023 10:57:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 42FED6B0118; Mon, 11 Dec 2023 10:57:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2DDDF6B0119; Mon, 11 Dec 2023 10:57:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 18E096B0117 for ; Mon, 11 Dec 2023 10:57:49 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id E70D780691 for ; Mon, 11 Dec 2023 15:57:48 +0000 (UTC) X-FDA: 81554993016.22.98E4CAD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf15.hostedemail.com (Postfix) with ESMTP id 24CADA0006 for ; Mon, 11 Dec 2023 15:57:46 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WQeemRLz; spf=pass (imf15.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310267; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JUV4YjA/avU6lZdC2uH9xVMs5FHkuyGC756X6BS6p9Q=; b=5r4Gm4QD6KGdNGoDeHjDQ/qJt6j4QDeqmaj6o4kL7wK0mwPGvJZK4yOXpUEtd/LDjji+MB cXM2VvlYRAky+hCMHC7eb95pwvdHUB7PWU0VxNTBYfhiiXpWFQx4l4ivFzVJPbl6hXwPQ9 e6no8cqOsoNwHCPQKiNcnQkzzcPTgzA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310267; a=rsa-sha256; cv=none; b=NXbK73XV1i9VShLmEqHA60QPver7Oc0nrvKUMFK6SXiLq/5iYUHm5FPD/E3apqEhAxMXDa V+Hq5n8iE1DVzcOO5wcJLvj/Dhw73NFSvCXzgbadFnt4w3fvC2l//5UyTqyy+79XOgMfhP wm3jWwGUcGW2/WVyMsoorKb+9pj9BAE= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WQeemRLz; spf=pass (imf15.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310266; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JUV4YjA/avU6lZdC2uH9xVMs5FHkuyGC756X6BS6p9Q=; b=WQeemRLzn1CudUMSDad25WhZtNalq2qADrM2qFwEksIiKgj0cqgFUAj0hgY/GDuKjuodZ0 /JDnbpPt7Pnnt855D26SIuYhNPTX4Jw4Jscy6ZAlB0+ukqzsM0uFmh3gCZaSxd2K14H15+ 6XmqW4wVTUp18hRtIsALBrmlFt0bO48= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-501-MIk6dcdIMbiRE_u7hAYpUg-1; Mon, 11 Dec 2023 10:57:39 -0500 X-MC-Unique: MIk6dcdIMbiRE_u7hAYpUg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 134DA870831; Mon, 11 Dec 2023 15:57:39 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 012041121306; Mon, 11 Dec 2023 15:57:36 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 17/39] mm/migrate: page_add_anon_rmap() -> folio_add_anon_rmap_pte() Date: Mon, 11 Dec 2023 16:56:30 +0100 Message-ID: <20231211155652.131054-18-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: 24CADA0006 X-Rspam-User: X-Stat-Signature: enkh4qupqx1brz8mwto51o1qqpq4j3nw X-Rspamd-Server: rspam03 X-HE-Tag: 1702310266-839492 X-HE-Meta: U2FsdGVkX1/tq7RsxQtcRPTn9x53xp7bqRFSjpY4ve36dwOHrKhh0QNKkzNeEPKPBgmJAUt9wdt7BzyS24mkl4zreJ6u+dinP0gkL5Zh0YDzpx18SRe32g8iQIRtDkvjXxwyyg6arB7g657vygFn8h4U431VXxJM4UQAw8NIBreuqaXG8+04f6g2VDpI3Csxl4ptVstyFNUZSn+H6G6DxxrglocsqepQXRZdMkavacUNjDq0g1Xb7pF2K8idF9nmFDijUjjNpkvBCK2qVmPDetyJfzqzg9AH+EOZqe4ql7Fsnmd3tcB9vJyjbcL8B9fTr4VSbPX/8jbTaLcawCya7iARNl9SIugtKzZEEyQyNwxP/EcYShGf1MmwrU0RpF400rXn4TAcpn9aH2kLonBzuh6kj5znN+qeGLQmrVQXcLGi+sIH6H4ltEzeAwktKsC1nxaS8+EZzuWfJmFAf0gjjqZbL0ITW+0cZ9JTAYDy7S499TyKlzSOAMGAASlLQp/kXShiz2zznNOsnWr+h05h4N2+o87omvSt2KoXuezdeB4psvzgXK2HFjkb2RAPkwlQbjW48LenGrL3tv1RX/krchsV8X1VB0OcrVmTFK+QL2I1Z/oYOgkVRtww5PBsDAon36Bfy8Iu79iuZYl4gC7wCR+wibgum9etfzfxt4bq9V+KWMGVtXsEBHjSm9G/e5Yq4gztCMov1t4rNrj3IJz8yzucPzhvzlDNXrbg8460p8/GMP6Z9zUuXX7ppYQyTxmAzO7PmG570vBOElMiTSJc9ujteTABxP24iHV7v0fEVrpsFFHuHNpUXVDSUkaKDgfQAG+jI3Hp3Umecre+BI4f8Sn2iKP+B1vJtxayQCQHSqVZvS2biwQsF+uzrjZSatyITI+dmX+b3pkv5CfTSmWcEH0qMTUg4VmBevQJJ4IKoSSfDLnCn2uL5c5dDaalhKGkaHGwdrBnQSsgqGbMAlw T5wlbTTN w5Mr+VFmld+SvzGX3Ol9CREmLhTHN9Ue8hstCCKTLqGrM6UXkwlYYYm3h69ywW0pmOfo2Mb9uj1OWN7Jejr5WJSsocGDoFAeBfd62ELKaNmvX4hMfT6ASzWQOOe6/iFOEG6jTS9TEmApDAqwTtnU8/NsPdHbBazyD4rrYm1aUhsziNL0n753AA5ztvrRwYA9xfAKSQ8DB5icDk3R0IWTvih734Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert remove_migration_pte(). Signed-off-by: David Hildenbrand --- mm/migrate.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index efc19f53b05e..0e78680589bc 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -259,8 +259,8 @@ static bool remove_migration_pte(struct folio *folio, #endif { if (folio_test_anon(folio)) - page_add_anon_rmap(new, vma, pvmw.address, - rmap_flags); + folio_add_anon_rmap_pte(folio, new, vma, + pvmw.address, rmap_flags); else folio_add_file_rmap_pte(folio, new, vma); set_pte_at(vma->vm_mm, pvmw.address, pvmw.pte, pte); From patchwork Mon Dec 11 15:56:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487459 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0EC22C4167B for ; Mon, 11 Dec 2023 15:57:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DAE356B0119; Mon, 11 Dec 2023 10:57:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D2FD96B011A; Mon, 11 Dec 2023 10:57:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA7C46B011B; Mon, 11 Dec 2023 10:57:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A335E6B0119 for ; Mon, 11 Dec 2023 10:57:51 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 81DFF1607A7 for ; Mon, 11 Dec 2023 15:57:51 +0000 (UTC) X-FDA: 81554993142.26.098F554 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf30.hostedemail.com (Postfix) with ESMTP id AF9E18001B for ; Mon, 11 Dec 2023 15:57:49 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OZB2h8yN; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf30.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310269; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6EJB9NGtrAo1f2+F9yoIp/qkcy9rFrccif86+rqYixU=; b=vjLzDHv+UxTTqEaSODEtIFLoxfopIjszMEf3vyQz1QWpKW7Zy77mj/ra0D2VJHM25KIQwV SDJApePI/1xmvGH4mGmb3vIWSLjDfBohZ/NdwgBDlAlNvrgsszTBOEip9O59LI9G3K8QXX +4xKt9AZoc+/4kyr6b1BqMWnCoAtV5k= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=OZB2h8yN; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf30.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310269; a=rsa-sha256; cv=none; b=Z76zrAmoKMCRIzp21c06kPrt+sPd4b22W2kZ7X16Mok/gTJ5kpPV2cOG2fDiSYxsvRf/UT UzlHGC8MU/L56Iilk0PDh7z3I/+HaV4L4CZ+aYkQ8nTW1JU5sS75sTSNz4DmSV5zbP7be8 FeryjSlfHEei4vAqKIkIFtGYweJrDaA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310269; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6EJB9NGtrAo1f2+F9yoIp/qkcy9rFrccif86+rqYixU=; b=OZB2h8yNR6wjBuya4G/w2HVruSvog7S6Z3Zq5ttjb0EY3haMICk+hPf7zV2INahKI0+UqO HuT7kkLy365cPFUROgP3SAEZGn2YP9pkrzfKMaoq91Q5MDMxJMVKTpPO/95ntdjx4jmmf7 0R3s+s8SX5hA7pbR58y7leuokn1TsZ8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-418-_8Xjipn3N8OVVf8cQocN-w-1; Mon, 11 Dec 2023 10:57:42 -0500 X-MC-Unique: _8Xjipn3N8OVVf8cQocN-w-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 9971B101E155; Mon, 11 Dec 2023 15:57:41 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 587D01121306; Mon, 11 Dec 2023 15:57:39 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 18/39] mm/ksm: page_add_anon_rmap() -> folio_add_anon_rmap_pte() Date: Mon, 11 Dec 2023 16:56:31 +0100 Message-ID: <20231211155652.131054-19-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: AF9E18001B X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 5a8gf6i5hnbf59yikzedhry41rgh7x8a X-HE-Tag: 1702310269-988896 X-HE-Meta: U2FsdGVkX19IvEaknkIbnKIma60CdBcS2YgFCM9+UXuBeJH+6Usi9mg2Eg+BiBRQiLQk00CKeFZK4I59nxjETk6iM9w6ZJkubbUtLNX/+zS73HeFXsI2u4+hREqORucCgjpYXem4bJhGWqBUcBYwlts3LOMmiirItm7VYgpvEy77oaYvtO9xGmvXPIZDC96t3RQbdd/3lOLh4D2OFrx7M0I8ubTvJShaQV/OPMxG6l6HnQdiv/oLJ2p1PbJLrCBbkBGGbm4GugmSQg8q82A+6hB687gZRHG7vyr7dilBSqGxYOYubH4CicDTkBc1ipnCnm/lWxjg8YEddV03FpYZlI8CKlgZVkwRQdsWi5R/GDC4ZdzIL54rGO4Ydw1NL6lBYGG5fndmr0agaGe7d/SIa3VtlR+TzzJOtkafw5pA0ZuAJtKVIFwfRqo13IwvN28kpYD2tv5zrNE8U2dTew7GeAZCRnL4no+OiZ6PchnK4C+d6tbcKjiZaE1lm4qH0McM90msavrOMgv+eOUkVDqoKVDc4Mso6xMklzXG2D4EFWMNw1XGkr09od+nAjyZMIDCfZDR3EzQQ6qI62Be47yCamFx9VW+6RY6lIZtXz8i069ZRtuqcNjf8mpTulk6ndox0/L+gkAurSJP1gUdRmk9ieeK1iIFrXijIxdLFu26xkf4vhORJQZVnVK2U7UVLXTsQIdKils8VPR6n/9idWmoJDiPXRMO5i/xM5tWQeGh8VOSUOj2dyVWqtPZHeuoUNxYtQY5FnhVoXVjPicacQn187ZwLdyV7bQTvCfsam8mEaJ0estX06FzE5WZR5tjii5mYheyhl2ry3GrKFDgyNBC7C2H3c53lA+3Z14G6Zo06pHnNd63su2+46ORcGW7DEDBKEFrIWy2EpDNvrpwv68Kq6gHfhQev21e9z68TOncSAddgwyiAwlglDiJ00MPjKo1Fs6CaPDHPKBIZg6SG67 M+K3LvI4 j6Nkp+3gllpSloXHi/3d9eNMVWR6iSW+ZaOTgVhyPKiz/GUm4wp02EzHPfnVMf2ac75c06UZzut7wG/MIxYHvRHzGSV3Y4/qAaxiJ+uOgBqOGmVGpdrHrr9tzn4he96UO/O+2mMDaBlI8K1Wb1BnSNaUg9SQaIq0FHKkQgQscERXH+5klaRMPNH7asXICYq6n343XXSbnD6pnRWFW9KJvz5WVlQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert replace_page(). While at it, perform some folio conversion. Signed-off-by: David Hildenbrand --- mm/ksm.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/ksm.c b/mm/ksm.c index b93389a3780e..2b6888ad1470 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1199,6 +1199,7 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page, static int replace_page(struct vm_area_struct *vma, struct page *page, struct page *kpage, pte_t orig_pte) { + struct folio *kfolio = page_folio(kpage); struct mm_struct *mm = vma->vm_mm; struct folio *folio; pmd_t *pmd; @@ -1238,15 +1239,16 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, goto out_mn; } VM_BUG_ON_PAGE(PageAnonExclusive(page), page); - VM_BUG_ON_PAGE(PageAnon(kpage) && PageAnonExclusive(kpage), kpage); + VM_BUG_ON_FOLIO(folio_test_anon(kfolio) && PageAnonExclusive(kpage), + kfolio); /* * No need to check ksm_use_zero_pages here: we can only have a * zero_page here if ksm_use_zero_pages was enabled already. */ if (!is_zero_pfn(page_to_pfn(kpage))) { - get_page(kpage); - page_add_anon_rmap(kpage, vma, addr, RMAP_NONE); + folio_get(kfolio); + folio_add_anon_rmap_pte(kfolio, kpage, vma, addr, RMAP_NONE); newpte = mk_pte(kpage, vma->vm_page_prot); } else { /* From patchwork Mon Dec 11 15:56:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487458 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C1DBC4167B for ; Mon, 11 Dec 2023 15:57:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 801D96B0118; Mon, 11 Dec 2023 10:57:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7B1A76B0119; Mon, 11 Dec 2023 10:57:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 653126B011A; Mon, 11 Dec 2023 10:57:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4F1A06B0118 for ; Mon, 11 Dec 2023 10:57:51 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 26775806DD for ; Mon, 11 Dec 2023 15:57:51 +0000 (UTC) X-FDA: 81554993142.27.DD7246A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf03.hostedemail.com (Postfix) with ESMTP id 5DB222001D for ; Mon, 11 Dec 2023 15:57:49 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=N8J4wquh; spf=pass (imf03.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310269; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=g4JhnXoUG0c2uPae4pqIp1MlWaLrLx7AL/tRs/hDXm4=; b=xldRb+fi8tbcFGwUFfon+EeAea/btmyvvmjPoRtdaaGtx4uaSS7XcIc394y0q3SU+h9dnz srDx2euhMS82u5N3szZrVImYqcfmrOLVGxjnJjVFI70/TB2YMECQszCdyIjkJqZSMggZL1 OqXVTtzTrmy5nemLIN92nPvv8xgYN/g= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=N8J4wquh; spf=pass (imf03.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310269; a=rsa-sha256; cv=none; b=VL1c+L/oAJdt1+/PuhTroYqjMI3mdr9qkoApkcc2gIInqe+2sxCe86fq+BxEiKXNHyfGCa sFlpRBJSqL0MHUgFoVMhb849KNS23SlvEEeiKca8Uwm9y+XkGr9OcH+znMPebhSDHGCyn5 cwpga5Z8Adr7DTLNwnzgi1J02lbg3x8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310268; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=g4JhnXoUG0c2uPae4pqIp1MlWaLrLx7AL/tRs/hDXm4=; b=N8J4wquhsajpoHQiXXHeT6Pkf6vghjAvA8D7EzYYWjaKc5lMh89bNsJQCXgdfpnMlKD+8q k71TlMTJFMpDZkKhstV7nuuOBICxeSLavLEBC6+yt9SwsQvtdJH5R/0TREM6m1rPyW4Mxw 4Bk63TrATiE8vhUWVIKrxoZyIUDhPMc= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-350-BCpQV4YyOqmsm67nADb-VQ-1; Mon, 11 Dec 2023 10:57:45 -0500 X-MC-Unique: BCpQV4YyOqmsm67nADb-VQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 918BA848A64; Mon, 11 Dec 2023 15:57:44 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 03BE91121306; Mon, 11 Dec 2023 15:57:41 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 19/39] mm/swapfile: page_add_anon_rmap() -> folio_add_anon_rmap_pte() Date: Mon, 11 Dec 2023 16:56:32 +0100 Message-ID: <20231211155652.131054-20-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: 5DB222001D X-Rspam-User: X-Stat-Signature: rk1y3rdxxr3ypse7ubfxzf3r7n1rtedo X-Rspamd-Server: rspam01 X-HE-Tag: 1702310269-278521 X-HE-Meta: U2FsdGVkX18BZsIHwvQVH9OoDqmvIN5ziGMAR3LN3E9fu8+3GXtRBCvbIyLVu0aKE28J5Cl7D+JfXH9etDPolhv/71+8ygIntsmJAujK7JY7z57rFS44XRQbH4velVQHmKyEYsdzgWMWHXZzHsG0AHVXmi9EpmTo8j0oZUjzl3ew2UHt9ty5eVj1Gan8xrEYZpKxg9mUjLTgRnaN4KgYvIWahXRxNarzIxLAOLNUQkISPzELI9UKkg02tP3PM6zBbpZamsTOBl4IJ3+zwNOv9WNjKGM4NrzK2DGZWP+B7nWJu0Y31HWJBX9s0aZ1PnaUMRXIhoOswswP5mC/ddceW2Sau1fam2wW2aXTUbr1Q7BkrWmJ5P4c01ZRyVX5LT6uYjbjL9e1h/9HHoUPpTzYU28z0n34fWCQSPXjVzqdR0j3VMGW0KGfj9/9/OvQUKuE/WehGj3fHWN1qOtvnE5UDtodOaUCioC0nSg+yPUFVGQIUfZcJ0Nq1h7L0HFQcDSW+lrruADjEOxXqw1KsPFy48WBcev/aOJYb/V0dd2/NVTAKbmbQkhqcq6vDjPvN37sdyXGUrPRzrUwtA2V1yO7rN3t3f46WSiNomwFI5L6MzQlX6dnG/8zdgEku9r9aHtdsKEOKfy5ZIAsdg2o9jnbAGOo1vqmIeiZYuR3xvnJG4tKw3HKlVCXmW0CWDOzP1u9R59WBlhw9IACtoshXuO7Rf2kwWytELS+ecTtvqN6PQWPvzo3ygZlcEHMnWw4Erdqs+MNjngEzi4d6Kx184v26ufLyTGqNeQsXxNSnn8sFY9w002LeEZxE7clTs4mVvupPmt3Wkbu5feAhVrmK5h2QzwHTnwH7iUpalpCt7a5RpBIUFTqPYHXvnPaNDSAjDAhX85PCOJ8uNMzbfNq01ugeHfL8I6h6EsbVuOWW8xX5gFkeflNx0YzzrJ6/FmzbWvk643V1Z+1dBXgD6TRN7w iLI1rd5A sno6LFrSbHf67hxKPfoO3y2HZoTPhxqUtRvsLfBckcCOOrwCTVYE8K9tNmNwBdRL5MII0fe50BuiuHVjTsKAcmQToH7tieMu0e2BwNsOqwfm/67a+rDaBA3gnhbI8nh5b+eXqjBmInUNPa2xF96K4TVTGyqbvBoI1B0zpfm6rLDgAuASBIFdicNYqgLfkpYfQlblMHq51aqcW+mSCsrfEWlkKsA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert unuse_pte(). Signed-off-by: David Hildenbrand --- mm/swapfile.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index 8be70912e298..25f53bec5097 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1805,7 +1805,7 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, if (pte_swp_exclusive(old_pte)) rmap_flags |= RMAP_EXCLUSIVE; - page_add_anon_rmap(page, vma, addr, rmap_flags); + folio_add_anon_rmap_pte(folio, page, vma, addr, rmap_flags); } else { /* ksm created a completely new copy */ page_add_new_anon_rmap(page, vma, addr); lru_cache_add_inactive_or_unevictable(page, vma); From patchwork Mon Dec 11 15:56:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487460 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3395C4167B for ; Mon, 11 Dec 2023 15:58:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 03B106B011F; Mon, 11 Dec 2023 10:57:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F2CC36B0120; Mon, 11 Dec 2023 10:57:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA82E6B0121; Mon, 11 Dec 2023 10:57:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C7B816B011F for ; Mon, 11 Dec 2023 10:57:55 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id ADCFE1A077A for ; Mon, 11 Dec 2023 15:57:55 +0000 (UTC) X-FDA: 81554993310.14.7BA7084 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf25.hostedemail.com (Postfix) with ESMTP id D9856A0027 for ; Mon, 11 Dec 2023 15:57:53 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CMHusMLW; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf25.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310274; a=rsa-sha256; cv=none; b=8jAlMXM1epXDvpvRem2HAL/tD1SmpMJhZ+3Ma1eGUSzjECLUbgEoRWcC0Bf+48TR2Jqwrs TYRNGWyPwB7ZXaLTy4e0ohST5zbuoyxREUH4ew6EmKhcr2hlbUzA/PDNLhjEhhZYvuzWa5 lg8kVidzDipX2tw2Yurvj9EFya2JUhQ= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CMHusMLW; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf25.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310274; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ErbCAGluk3VXsqVDEiyD10ub89UxQEpB3+XojBc5SDY=; b=mcFi6GtxzodoylU2wDb9uBEXMOMFY9gd4tl1D0K9DQPJFAIuRXmftP1hIcL/3kxGoTrYMs FGNSLCMqrbM6Ik44+NJW9ey3hW4S+tiCpNLOTVIAQl260qYZVl0oTV4OgQ7L67Uf/OP2wd hC78NMPYP8axzVroIMytz6KH1lMOR4Q= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310273; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ErbCAGluk3VXsqVDEiyD10ub89UxQEpB3+XojBc5SDY=; b=CMHusMLW3UDXs2qo5zgCdH4m6KCKMr3eejdr2bslvcApqNQk6Q1Z8LXi35rDKnn+eHMLfX Eqh520jsdj9HWNZjnBFwyuKKxYNihMmOee2oBMU8UU8MTvkw/nkwyacER/oAqSFufnIMRd s4a6yFci/8fB3xURlQu0/lcxZC7zZOM= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-690-mgQppAEyP5iFD7nuwA5GFg-1; Mon, 11 Dec 2023 10:57:47 -0500 X-MC-Unique: mgQppAEyP5iFD7nuwA5GFg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 57849870831; Mon, 11 Dec 2023 15:57:47 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id F2D511121306; Mon, 11 Dec 2023 15:57:44 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 20/39] mm/memory: page_add_anon_rmap() -> folio_add_anon_rmap_pte() Date: Mon, 11 Dec 2023 16:56:33 +0100 Message-ID: <20231211155652.131054-21-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: D9856A0027 X-Stat-Signature: 7p9cwmirh6wouk6bfhsh9b338ksyc3rh X-HE-Tag: 1702310273-468386 X-HE-Meta: U2FsdGVkX192vqN0fWPqSMvUtKCBWK/nwZ1xxgv/J5xvuzWPc1iw2rSXF6qfG04Z6yd6mwVzlJKqmh1ClifVNIESIzi/EO6is8DjLlhxshRvrT675/dImOGROYvXKvVD5n2z1HTN5BYPmBP6HFhYhqgbKiahRBc/eT6FzTpWq8PPYi9ZyePbAyknoC00MdXmUGmNLzE0Ugn6I9hdAEac/20L4WgsvQBqAYPnNohFqG6VH09lFt8K5T54VE825LBbnwFrgwPfdu2/RE9hS0sRMeROkGHYjTBLxB2kgyVc1Ci082cw6zvbU4AgLadwrFjHrVwOneWrn/XPuiUoSBdLAHp6wA8bICEmKUtI1q+iGb5LmV60KYPZPMRtLe5MfhN/+bo5c1ryd4CHBhSJEF4sPRXiBToR6w255xQIGdGYYVO1CzsqSGGWNlc3t7ces6cWxM+hZCC+YOVxVpA/b9Piak243W9GO8FcjtKzjoiOIYZRININbJ796xg0suY/YCManwQXYvWJJI7anUZwL78ZmBykK5IFIAB7F8xx83F+pyr9+r4ZtbDWtqn6hiMqt2Rft0Yr5uKEOaahbqm4AlaAOvtz2q0oJ2f4Iup9Bpp1wRQgsRBZImDB2pRK6sTRNSGXVmyG+Klu5bbufnA0gCGNN0O31wsEbx0aAIfjo3s4uNjr4OMwhXBtrL3IOMKU6vYr3VM2DQqgmO3R/E3/s1tOqv0XPzApNcqhuZFEa9GxoCwjPQUSIR/F4N25Ktq8LLb1rtsPf39JHCSX4eRnofVBBdWbsigoewCsf7z2nzIJBdF1p3jBIhAbwFNJGhZv+YrFSsJmXqV8GZ6MtNzG6lGcoveA7DrUoEA3TiCEse3/cT1CnuIE0YDu2VZ64rUEHMmsSUS/cIPea8lOx0RRZfXsr9c0fch+H7G/DeloEVOBtpS5fWAvZUsYmLhSDOjE4Vi6eNL0euiBpNBJEtvGjc4 HTqk/B8o 0usGOtrZSm5P7JB/w652fr9kzR3GPCtHKF8T/kVS1k/bVSsCsNYlQjS+yX+2TgXBIwVPgBf0Gu9J9F5oAtHtv9bc34SUKXaW4d4+2b6FhyNhiI/6Njv2+I+R2mY+9zAZMY1SglJDaOtZr3d5AIJA7DNZi06n/AaJ8gpBBvu+Vg/9lqp8C6ksa5R4YWwQqMCEyLyrfud4npwh0hMl35/liwYMPfA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert restore_exclusive_pte() and do_swap_page(). While at it, perform some folio conversion in restore_exclusive_pte(). Signed-off-by: David Hildenbrand --- mm/memory.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 70754fd65788..97e064883992 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -710,6 +710,7 @@ static void restore_exclusive_pte(struct vm_area_struct *vma, struct page *page, unsigned long address, pte_t *ptep) { + struct folio *folio = page_folio(page); pte_t orig_pte; pte_t pte; swp_entry_t entry; @@ -725,14 +726,15 @@ static void restore_exclusive_pte(struct vm_area_struct *vma, else if (is_writable_device_exclusive_entry(entry)) pte = maybe_mkwrite(pte_mkdirty(pte), vma); - VM_BUG_ON(pte_write(pte) && !(PageAnon(page) && PageAnonExclusive(page))); + VM_BUG_ON_FOLIO(pte_write(pte) && (!folio_test_anon(folio) && + PageAnonExclusive(page)), folio); /* * No need to take a page reference as one was already * created when the swap entry was made. */ - if (PageAnon(page)) - page_add_anon_rmap(page, vma, address, RMAP_NONE); + if (folio_test_anon(folio)) + folio_add_anon_rmap_pte(folio, page, vma, address, RMAP_NONE); else /* * Currently device exclusive access only supports anonymous @@ -4073,7 +4075,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) page_add_new_anon_rmap(page, vma, vmf->address); folio_add_lru_vma(folio, vma); } else { - page_add_anon_rmap(page, vma, vmf->address, rmap_flags); + folio_add_anon_rmap_pte(folio, page, vma, vmf->address, + rmap_flags); } VM_BUG_ON(!folio_test_anon(folio) || From patchwork Mon Dec 11 15:56:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487476 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2C20C10F05 for ; Mon, 11 Dec 2023 15:58:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 458776B013E; Mon, 11 Dec 2023 10:58:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4096A6B0140; Mon, 11 Dec 2023 10:58:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2344E6B0141; Mon, 11 Dec 2023 10:58:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 0BB416B013E for ; Mon, 11 Dec 2023 10:58:34 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D3693806DD for ; Mon, 11 Dec 2023 15:58:33 +0000 (UTC) X-FDA: 81554994906.20.3C86F1D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 544CE14001E for ; Mon, 11 Dec 2023 15:58:30 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cLLxRgvn; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310312; a=rsa-sha256; cv=none; b=FR+wj0Z4U5wWkzndyEDqzDAXegO4lQSQ4gYpv8UlCqL/n9gQhyoMt5W5n+IL3/JB96u9Xr /TPFTwWm26qZ89HF+YgoLqeb1Fdov0qGd5YEcibuSz/SDr+LJ1Iaiiwsn3Use4bOjW9ghT eycbN8DWu4m3NQjZ+2C7y3qmgONDqd8= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cLLxRgvn; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310312; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kqof2lbc18NuxxZ3avfpRF1Wh/FjZz2qWodEICCpfdI=; b=NnJAXcuctu9X6kQ5soiEfonZP6uYPKi8AbF69C2P7SjtVqWaZ3ga6Y41d/DKOqrJyEuixD ZZNPg6HQoG/TDmWWI5g2dSaUi5bWus3xn5ecd8ThHrOVT4RVzf3iJm0OLj98fKI3H515tE Rl03U2vAmXbYRshI7zjwXIAlBUCHPlc= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310309; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kqof2lbc18NuxxZ3avfpRF1Wh/FjZz2qWodEICCpfdI=; b=cLLxRgvnoGGkk1gtRoyhot30SqV6aaygZT7aQGC2mZ7dFRPfA0ZaK+dv8uABToV1YIwFvo yNJLck8AVrA1Zu7KNIqRwyovKZpUeALNqfpld9C8bJp/SOkMJFzKJLWqBlYrDhisU2L73h TY/65dy8cX6TsAGupuQhr7OZ9Vbb7ok= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-146-nQ4jaL5vOtq2vnJawkD1nA-1; Mon, 11 Dec 2023 10:58:04 -0500 X-MC-Unique: nQ4jaL5vOtq2vnJawkD1nA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E265A85A58A; Mon, 11 Dec 2023 15:57:49 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id B71401121306; Mon, 11 Dec 2023 15:57:47 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 21/39] mm/rmap: remove page_add_anon_rmap() Date: Mon, 11 Dec 2023 16:56:34 +0100 Message-ID: <20231211155652.131054-22-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 544CE14001E X-Stat-Signature: 6s16aycyqs51kr6fa7mgeu7z5e13rumu X-HE-Tag: 1702310309-12435 X-HE-Meta: U2FsdGVkX1+Sr0ZDhxqobpjoNhGtDdSCNWJRZmCn7K+musGWtmvx7OhCHM7dOVEXI2ND31XN3ApglTfRH78O5Meoqb82ZMmeLy62wGFlHdNvLLZb4rxl4pEhI+XbsV9oyihcKDD6VQxXR+g7aAS9NiuWRPesIjLk1qV/6dhI5Y9MINyDJPcTVCctIjxoN/a4qUyp77BDK1dV74RU9qXoYtZVX2UAQYK5cm54m4OUScqlxMPponusnDA69/E31qfpfYOeZylW5+EIFuopDbtd7suNtZtaVkIcKMS9krqxQ70i65n+pupqp+e7+orvD+ER6kGC5jIs54XM8upBPIvFEuxi/OZM3VUueQ6i8Ewu3qn4Go3/Cp1qHNC2ItkCePBL7PWs6PXKciO6/WEqJXIRdxVffXq9GSVYqBZjQe7PHI3tJCtvtM1cvVXEtiwTwUB8ZlLikVQ+E4Zk8WuTiTUZlWLs+9vby9AelLsubNAbDZCByHEYIMmZdhnGe7DPPuFHdg8T8uV1Y54ShaldxgQU0tVunlxVU2S3e3LKEux4F60SImsyOJg17mp92MEYvxt1SzCnAVFaEhwHscJeHL6uGue0nFLGt6w9KCLtkTCNPbiHm4i4nUJUt3gPm9FJOheB33wTNWUpBknsfq10thnfZK6mch12x9433boP16nJEek5muX0Mm1+McIallJCnF2NtPRez3cAZgId4c5Rmu2JhKbK29ui2DA7DIxayfzWEuqAAN5D+XafULJ4IgsRvnADXHjVvAy3BkuZwCVGO2FIx8+TLv9Hexh2NmptXW5xiA1CtIZVkZC/o3XY88uJQGAHx5TnSkDiYo5rVmMDgHL7uK1hY6O224Mi57LkHpR8ecXtjXw4+oMVtqWVcderEaKCirapJXKGuB7nllseQCu98Jo9Ap1vAohnF1/F0Yynxu83GNPosMQP9DQ2ShHfpULnSiFs4GtnF6zoVBi4aEI lXYg/FGA sfs4Itee7pMpoKVSNvE+luPY+P05S89hJ8Y9ruoUxQLDma0Lpi1kGITW1XSaECyo9DbKzyR73Q8nFGj2QD61KXY03FuGjJZp+b74Mp1p1AwmqAoWvE3wbdUYlNcsq+ertp5xpye9XyRGVCr3OG9RuGwKneBHQ5gVz8q3n791DXzJd5r16FrisMtag+qgxAR/i6fuV91HGJz+PUBNw/yZfHdLp1DWrDevquETP X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All users are gone, remove it and all traces. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 2 -- mm/rmap.c | 31 ++++--------------------------- 2 files changed, 4 insertions(+), 29 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 3b5357cb1c09..bd4edae4dbe7 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -240,8 +240,6 @@ void folio_add_anon_rmap_ptes(struct folio *, struct page *, int nr_pages, folio_add_anon_rmap_ptes(folio, page, 1, vma, address, flags) void folio_add_anon_rmap_pmd(struct folio *, struct page *, struct vm_area_struct *, unsigned long address, rmap_t flags); -void page_add_anon_rmap(struct page *, struct vm_area_struct *, - unsigned long address, rmap_t flags); void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address); void folio_add_new_anon_rmap(struct folio *, struct vm_area_struct *, diff --git a/mm/rmap.c b/mm/rmap.c index 7787499fa2ad..83cba8909848 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1271,7 +1271,7 @@ static void __page_check_anon_rmap(struct folio *folio, struct page *page, * The page's anon-rmap details (mapping and index) are guaranteed to * be set up correctly at this point. * - * We have exclusion against page_add_anon_rmap because the caller + * We have exclusion against folio_add_anon_rmap_*() because the caller * always holds the page locked. * * We have exclusion against page_add_new_anon_rmap because those pages @@ -1284,29 +1284,6 @@ static void __page_check_anon_rmap(struct folio *folio, struct page *page, page); } -/** - * page_add_anon_rmap - add pte mapping to an anonymous page - * @page: the page to add the mapping to - * @vma: the vm area in which the mapping is added - * @address: the user virtual address mapped - * @flags: the rmap flags - * - * The caller needs to hold the pte lock, and the page must be locked in - * the anon_vma case: to serialize mapping,index checking after setting, - * and to ensure that PageAnon is not being upgraded racily to PageKsm - * (but PageKsm is never downgraded to PageAnon). - */ -void page_add_anon_rmap(struct page *page, struct vm_area_struct *vma, - unsigned long address, rmap_t flags) -{ - struct folio *folio = page_folio(page); - - if (likely(!(flags & RMAP_COMPOUND))) - folio_add_anon_rmap_pte(folio, page, vma, address, flags); - else - folio_add_anon_rmap_pmd(folio, page, vma, address, flags); -} - static __always_inline void __folio_add_anon_rmap(struct folio *folio, struct page *page, int nr_pages, struct vm_area_struct *vma, unsigned long address, rmap_t flags, enum rmap_mode mode) @@ -1420,7 +1397,7 @@ void folio_add_anon_rmap_pmd(struct folio *folio, struct page *page, * @vma: the vm area in which the mapping is added * @address: the user virtual address mapped * - * Like page_add_anon_rmap() but must only be called on *new* folios. + * Like folio_add_anon_rmap_*() but must only be called on *new* folios. * This means the inc-and-test can be bypassed. * The folio does not have to be locked. * @@ -1480,7 +1457,7 @@ static __always_inline void __folio_add_file_rmap(struct folio *folio, if (nr) __lruvec_stat_mod_folio(folio, NR_FILE_MAPPED, nr); - /* See comments in page_add_anon_rmap() */ + /* See comments in folio_add_anon_rmap_*() */ if (!folio_test_large(folio)) mlock_vma_folio(folio, vma); } @@ -1594,7 +1571,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, /* * It would be tidy to reset folio_test_anon mapping when fully - * unmapped, but that might overwrite a racing page_add_anon_rmap + * unmapped, but that might overwrite a racing folio_add_anon_rmap_*() * which increments mapcount after us but sets mapping before us: * so leave the reset to free_pages_prepare, and remember that * it's only reliable while mapped. From patchwork Mon Dec 11 15:56:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487461 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB2F4C4167B for ; Mon, 11 Dec 2023 15:58:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E69FF6B0120; Mon, 11 Dec 2023 10:57:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E1AC06B0122; Mon, 11 Dec 2023 10:57:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C6DEE6B0123; Mon, 11 Dec 2023 10:57:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id AF52B6B0120 for ; Mon, 11 Dec 2023 10:57:59 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 886501606A7 for ; Mon, 11 Dec 2023 15:57:59 +0000 (UTC) X-FDA: 81554993478.02.30863C5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf19.hostedemail.com (Postfix) with ESMTP id A58611A0025 for ; Mon, 11 Dec 2023 15:57:57 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HVGBbowy; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf19.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310277; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KuC5xwuqcP7rQWdUGs+5SfITvp1Oi5oe13QDRULS4WU=; b=0ftAG6afFpt3HpjTXfdFjcm30UZ3WmBeHUs7kWuws1RtN6nyFiYIAw85bxDWmxCwGNKCQd iYhE45669y18NUez/gv9bdhCatA36EIXdHTBb/qDBnj8xUeiWGdxsgaERWb9WZrMkFsOxb wDTTw8JmQ4u2LtiafwyHKiP81BNo0l8= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=HVGBbowy; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf19.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310277; a=rsa-sha256; cv=none; b=6WLnvnFbs8tVQQQomN8dCWyuerdso3Dcj0fJxjDSZ0icgLoz20qhkmCZayHRice+6jIS1t E+XjrYhWBdGTmY4UiRCVH9jtm/m9HHImyDyEbAwR2jvgwsUinOLhKUEJtv+N++lsoH2/Jf yjtuSH4IMvPOJ0DVToMpCkG9nkV7J98= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310277; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KuC5xwuqcP7rQWdUGs+5SfITvp1Oi5oe13QDRULS4WU=; b=HVGBbowyJeZXhiFjYSfHcHcDig0oaeQOMpoMNqRNrvNHxB+p+gOi9qcEBWMvgX+SpuV5M+ WgSGu556vNpqZoqW8PeSsNg5VXfCYCSp9Yutb4mMkrPJHu4tbclv5Ae6YqUoH2ZrmHEEUR hY3tMyiDlb/3GyuTsSmjHhA9sy0fdC8= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-473-hqCqiiJtNeeYflAwvpNPtw-1; Mon, 11 Dec 2023 10:57:52 -0500 X-MC-Unique: hqCqiiJtNeeYflAwvpNPtw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D8E6585A597; Mon, 11 Dec 2023 15:57:51 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 25E491121306; Mon, 11 Dec 2023 15:57:50 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 22/39] mm/rmap: remove RMAP_COMPOUND Date: Mon, 11 Dec 2023 16:56:35 +0100 Message-ID: <20231211155652.131054-23-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: A58611A0025 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: p97x53yr6h8whc4fh3h1bwxy4494xw44 X-HE-Tag: 1702310277-504957 X-HE-Meta: U2FsdGVkX1/UeN3dKmsCREl2pUPwFNLcgIVbB+Kac1P/5PtwEXde8S1u8L8ONrCVgVkPLxyatDeZ2Ar/ljTeAgjaHtAceOVZBQpy7GCmNb2FrJqDAKOwgRV8CeOB4BVcMUfOspnJfjm2du7xN8dWONMOKfvvh7YeUFkCpi+Bf6yAYracNBhOWUhPvExwyt0tsIeZC8T8r93uT5pAAhb+UbNl75RrTfPAc8lRU1WYAQzTFp4DAJyOFpPKBQR89u8XhzKB6NHWkH8WChI172ZIVwEGbaaKNXZqyDNWDNYLKvMxuY8guelAjcLPDe8KvIP9Hdw3UuvqIuom7qFb2Nv6HieY9X9SDOrogw3DukMnqDAzHW34UFh8p5oAKKA+D6zQKggKcFAITeoSSFy+H11ZStTQCfdVeNnz/8P04n/lGKai+0zZra41xdtahJZcYrCaJEZrUcpS6iF1ZuysuHMALK8y8HeYPk25Uy7crdsqhuUyJdXPmBAvCIoLAaHlbPgfsQJBWiJOZkNPSAWuvlSo8mGEHciTRgY/USEyGkVv+ZSiR0ckdS5agd1YOZVbRUkYgVYyPQhpzo9cujfleb4Q6esSiMdlro5nXL2QtRuGGm76Askf8uVfhdR9ix3nst6W/Cp+obef6Itt5rK3zKLhJIxhVso2Yl90rx2Og4hLjfLa2XGWDcHEbpZvQkoHZ7BvmVRJ30N32FKID9NG260hnYwG/6YYsJpC/CoE5z91LlqO4MWLgE0z8LxuzQ8EHemDESaGs87R6C1tl52+LqCuow7Q35V7lFw8mn9Kykwzhotmha9tN7HPnLyhAXxbqHgxX61PCJ98tjsV2uN1KgjwUNBsjGL3hgjpK2kO0mqddgkQhKK3PeiMW9Sei86OfBYfcpZlxyx1VnU+zQQUN66D1sha+nbdo7s2SjI5yRV0ZAVgGaR7DXo0shXlQ9RqUPvILbGdIlTOgooYZzy+NXi QiQlRE05 ReWwiIwPlNnMl8e3u3x1SterRY8WWJj/2+zFoK4kFf5859c7cbdTuBW8OQgfBcD1+NZOTxlkoWHRhpVHZf71AZE2dEMNrf4QwiamXK7qJkTJBZaexoNh2AkQ0zaN98z0Kx8xANTsHqDpQvp3L0/jZW+ZSa6o2zbn7p1b6mhoLZ5MPTcX/I0KNQQ5PzuAOkt965MGGVK3lIxR/y3U/uLSMPXYO9g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: No longer used, let's remove it and clarify RMAP_NONE/RMAP_EXCLUSIVE a bit. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 12 +++--------- mm/rmap.c | 2 -- 2 files changed, 3 insertions(+), 11 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index bd4edae4dbe7..0acebe41ab8e 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -177,20 +177,14 @@ struct anon_vma *folio_get_anon_vma(struct folio *folio); typedef int __bitwise rmap_t; /* - * No special request: if the page is a subpage of a compound page, it is - * mapped via a PTE. The mapped (sub)page is possibly shared between processes. + * No special request: A mapped anonymous (sub)page is possibly shared between + * processes. */ #define RMAP_NONE ((__force rmap_t)0) -/* The (sub)page is exclusive to a single process. */ +/* The anonymous (sub)page is exclusive to a single process. */ #define RMAP_EXCLUSIVE ((__force rmap_t)BIT(0)) -/* - * The compound page is not mapped via PTEs, but instead via a single PMD and - * should be accounted accordingly. - */ -#define RMAP_COMPOUND ((__force rmap_t)BIT(1)) - /* * Internally, we're using an enum to specify the granularity. Usually, * we make the compiler create specialized variants for the different diff --git a/mm/rmap.c b/mm/rmap.c index 83cba8909848..9212726268ba 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2663,8 +2663,6 @@ void rmap_walk_locked(struct folio *folio, struct rmap_walk_control *rwc) * The following two functions are for anonymous (private mapped) hugepages. * Unlike common anonymous pages, anonymous hugepages have no accounting code * and no lru code, because we handle hugepages differently from common pages. - * - * RMAP_COMPOUND is ignored. */ void hugetlb_add_anon_rmap(struct folio *folio, struct vm_area_struct *vma, unsigned long address, rmap_t flags) From patchwork Mon Dec 11 15:56:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487462 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DC2EC4167B for ; Mon, 11 Dec 2023 15:58:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F9236B0122; Mon, 11 Dec 2023 10:58:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 284E56B0124; Mon, 11 Dec 2023 10:58:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0ACC36B0126; Mon, 11 Dec 2023 10:58:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E78F46B0122 for ; Mon, 11 Dec 2023 10:58:01 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C98811C107A for ; Mon, 11 Dec 2023 15:58:01 +0000 (UTC) X-FDA: 81554993562.10.ACB42AA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 0A9DB140004 for ; Mon, 11 Dec 2023 15:57:59 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cNvtRwPL; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310280; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OcVx4x0aUtY07dVPvydbU+uc8icopPtFroqSSpn7QWM=; b=cTwcAFoGAsY6pHA9sSq+YOSJLDpsaTglFP/5vI+sWahMBw6jhwf+sN5YGId7UZQgKwyzn4 kmkkC/CnFrZFL0mpEP5m9TjTtqUo1M8CjDiuASjGUX5zXXMEA9ketH6CZQ5ErkgV0U6k7s lh5SK+rnk7FS08cupZsnAwv9m6HFCIE= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cNvtRwPL; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310280; a=rsa-sha256; cv=none; b=U9u4nXQLHXJeieLGi0JJBzmDSXWcgr1s56Jcla+t6IjpoSAJkERMyht6Egdlvjxai3XS+r cCLncQSNPgquXUJ242I06BHaqNrDboMEJZYVCqAucjlaZThAXKeePG3ek4pi0wJ1/urKLU LscBo17ABBP4IlglxfUYEt/gk04AZFg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310279; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=OcVx4x0aUtY07dVPvydbU+uc8icopPtFroqSSpn7QWM=; b=cNvtRwPL3eFiW9oT7XnqoauygX8Wp/nkgWCFayufGsupZJlWy+PWrqNt8WWjAbuFWlWiQj Uvar4uVNKuSJn1s0R+dmBV8X38cVEvbZ8mgQoPxH6uB3pFkY+88RPKgD26QJr6NBvsLVeh bIgjFw05c0IQb4Dv6CyMaklC/aU+hZo= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-177-0UIjTO6mNmu_u3e5pNi68Q-1; Mon, 11 Dec 2023 10:57:54 -0500 X-MC-Unique: 0UIjTO6mNmu_u3e5pNi68Q-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3BFCA1C08B9E; Mon, 11 Dec 2023 15:57:54 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3E8C71121306; Mon, 11 Dec 2023 15:57:52 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 23/39] mm/rmap: introduce folio_remove_rmap_[pte|ptes|pmd]() Date: Mon, 11 Dec 2023 16:56:36 +0100 Message-ID: <20231211155652.131054-24-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: 0A9DB140004 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: ny9qganiojg1t88o3i9wmo41a97kddzk X-HE-Tag: 1702310279-791488 X-HE-Meta: U2FsdGVkX18BkF8Jy6zRLZzZT0CurTMFvliD89GhUbKLQitpkXxG4IZRKkzvuXrGZVtqzpjKjTu+Dp+ARV05cQ8pxQ3H9cAwYljOYD6a/VW8XEfNkEmX7D5eEkiLLX7KnA2lYGSunJxORWhtboU5ecTTBQ6iM96CenAzlOzqEJap4ZtkGUyLxTbKvmHcBFNyiRgXxNPMbSzim1I4EMsbftglgVqzOO67NkUxRPJZT8nHI2BwHZvH/hy+dee+Ja+Fc1yhhZiyEv+IbBCPTYJA6PURb2n8gyPqIn1E8Ydaw1m0z8H4zVyzASY15tIufx1GZO7wUSSF2RcJlIgU78RX3/YaIr+Eyrxl5aFmQ5pQawMirRXZ9BfdSQMGOXx8He/zevDNJ2cADvfv3g3sLtHfZyeYUs/GtSeBMdkacujzDpFHz65dpZ6Dk3dmaQMlYBn4ghMQnp6dCv4o+JD3ZvI9QnkzZOJ1Ievs1mVhItOLHjfN9nlGECpLCovCNycfOpy5nI6tq7514tNE7RmMRroKywjbQaX6ZMjOuuoSUr/g9n2hPyMsFOUvpx4LHwMKIFx4As3xPXB2qn21i8CfgWFygE57fxQ3RyzFYzYHPs00GMjzdCw/Twlc6rWZbARiRHLGyGONFHNrzuYH5cjtdjh0ZNfBuwtWSFIhv3FffZm0MANWUArPbS4MaiHeZNGqmm0DeY/nhyzRrawfFW/GrVqTH7GoAkWjIcLjMQko4fgCFM4CCXEuPTsGD0YbLUnsGgNttMCzZqsWV2uiJShHeAH33bFjLEsB9NmHmasYCKafHFD+33zHP+bonxEbSlNzmueYsAIUiC4/28mGZmxDPn9MK7qS/hdLsHRHW76K8FQT2SludrfkCgYsBbo4Q1Pr9cL4qZDlgDVJGfwKitHkNLh0QgXjQqlQXU0hWNmVU3OMnClpMC/2+pL0g1XfSu5MQ983TnKVMIIZ8ufEgzdq9i4 bbKza0OQ 4sPMusy/fotEaVE2QgfBicIIJrVjBasNRKyMHfZzbiDRMv/xKiJczDM99yj9qbAzW6je8D2idGUdgUMnUiJs065pAh5v7CF9P9MR+cj9r/KJ0xmp6nKyyylJ+8/5hTSH1zs5JVzT5T2LyFFLLBdRRqLfH1sQao7SS5GumuOhKcpE8OQBXzGR+2nEDJBrgW6m2cys+B0dl2S98CBhX2LnM6KIzx52BK/IbUrjN X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's mimic what we did with folio_add_file_rmap_*() and folio_add_anon_rmap_*() so we can similarly replace page_remove_rmap() next. Make the compiler always special-case on the granularity by using __always_inline. We're adding folio_remove_rmap_ptes() handling right away, as we want to use that soon for batching rmap operations when unmapping PTE-mapped large folios. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 6 ++++ mm/rmap.c | 79 ++++++++++++++++++++++++++++++++++++-------- 2 files changed, 71 insertions(+), 14 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 0acebe41ab8e..a266dc0ef99e 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -246,6 +246,12 @@ void folio_add_file_rmap_pmd(struct folio *, struct page *, struct vm_area_struct *); void page_remove_rmap(struct page *, struct vm_area_struct *, bool compound); +void folio_remove_rmap_ptes(struct folio *, struct page *, int nr_pages, + struct vm_area_struct *); +#define folio_remove_rmap_pte(folio, page, vma) \ + folio_remove_rmap_ptes(folio, page, 1, vma) +void folio_remove_rmap_pmd(struct folio *, struct page *, + struct vm_area_struct *); void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address, rmap_t flags); diff --git a/mm/rmap.c b/mm/rmap.c index 9212726268ba..dc3be5807cee 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1511,25 +1511,38 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, bool compound) { struct folio *folio = page_folio(page); + + if (likely(!compound)) + folio_remove_rmap_pte(folio, page, vma); + else + folio_remove_rmap_pmd(folio, page, vma); +} + +static __always_inline void __folio_remove_rmap(struct folio *folio, + struct page *page, int nr_pages, struct vm_area_struct *vma, + enum rmap_mode mode) +{ atomic_t *mapped = &folio->_nr_pages_mapped; - int nr = 0, nr_pmdmapped = 0; - bool last; + int last, nr = 0, nr_pmdmapped = 0; enum node_stat_item idx; - VM_WARN_ON_FOLIO(folio_test_hugetlb(folio), folio); - VM_BUG_ON_PAGE(compound && !PageHead(page), page); + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); /* Is page being unmapped by PTE? Is this its last map to be removed? */ - if (likely(!compound)) { - last = atomic_add_negative(-1, &page->_mapcount); - nr = last; - if (last && folio_test_large(folio)) { - nr = atomic_dec_return_relaxed(mapped); - nr = (nr < COMPOUND_MAPPED); - } - } else if (folio_test_pmd_mappable(folio)) { - /* That test is redundant: it's for safety or to optimize out */ + switch (mode) { + case RMAP_MODE_PTE: + do { + last = atomic_add_negative(-1, &page->_mapcount); + if (last && folio_test_large(folio)) { + last = atomic_dec_return_relaxed(mapped); + last = (last < COMPOUND_MAPPED); + } + if (last) + nr++; + } while (page++, --nr_pages > 0); + break; + case RMAP_MODE_PMD: last = atomic_add_negative(-1, &folio->_entire_mapcount); if (last) { nr = atomic_sub_return_relaxed(COMPOUND_MAPPED, mapped); @@ -1544,6 +1557,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, nr = 0; } } + break; } if (nr_pmdmapped) { @@ -1565,7 +1579,7 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, * is still mapped. */ if (folio_test_large(folio) && folio_test_anon(folio)) - if (!compound || nr < nr_pmdmapped) + if (mode == RMAP_MODE_PTE || nr < nr_pmdmapped) deferred_split_folio(folio); } @@ -1580,6 +1594,43 @@ void page_remove_rmap(struct page *page, struct vm_area_struct *vma, munlock_vma_folio(folio, vma); } +/** + * folio_remove_rmap_ptes - remove PTE mappings from a page range of a folio + * @folio: The folio to remove the mappings from + * @page: The first page to remove + * @nr_pages: The number of pages that will be removed from the mapping + * @vma: The vm area from which the mappings are removed + * + * The page range of the folio is defined by [page, page + nr_pages) + * + * The caller needs to hold the page table lock. + */ +void folio_remove_rmap_ptes(struct folio *folio, struct page *page, + int nr_pages, struct vm_area_struct *vma) +{ + __folio_remove_rmap(folio, page, nr_pages, vma, RMAP_MODE_PTE); +} + +/** + * folio_remove_rmap_pmd - remove a PMD mapping from a page range of a folio + * @folio: The folio to remove the mapping from + * @page: The first page to remove + * @vma: The vm area from which the mapping is removed + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock. + */ +void folio_remove_rmap_pmd(struct folio *folio, struct page *page, + struct vm_area_struct *vma) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + __folio_remove_rmap(folio, page, HPAGE_PMD_NR, vma, RMAP_MODE_PMD); +#else + WARN_ON_ONCE(true); +#endif +} + /* * @arg: enum ttu_flags will be passed to this argument */ From patchwork Mon Dec 11 15:56:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487463 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BD3FC4167B for ; Mon, 11 Dec 2023 15:58:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 599A46B0124; Mon, 11 Dec 2023 10:58:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D2826B0127; Mon, 11 Dec 2023 10:58:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 14ABC6B0125; Mon, 11 Dec 2023 10:58:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 010FD6B0124 for ; Mon, 11 Dec 2023 10:58:01 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id C70A9C075F for ; Mon, 11 Dec 2023 15:58:01 +0000 (UTC) X-FDA: 81554993562.13.CC74FAD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 0DDB240020 for ; Mon, 11 Dec 2023 15:57:59 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=G9uy7rYT; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310280; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jBiuDcFai57ywCoGHCEF1oxPQjyCBa6ek76+fUhJDqY=; b=4MWwGtP0nv8x/ZK1P3qJg8p/4dpfcjUM9GRBEjLN3+zge3xfAW3gm8aDDLPiUMARkpnc29 jBwOtCxFiwcKOTHjYg+SUy8kKvVsxabPINTSYlc8nVmAoySdMZUikcArsyWORDjAb6reES bWiVDOePaBhYWVCh+p7NvRTxizd1riA= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=G9uy7rYT; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310280; a=rsa-sha256; cv=none; b=DbbSOY8ORNbRk9rhTYjJLc9ZkQo/7KbAM7JxIRtzN2UKH2lJ+/EvDjwTjbBfUfDXTz9dCw jOMJswrnzXsJh4FMBBGeOEDPc5Cwgi/k/bBWQwwtQ4wh4ZO5oqrDqYADXOauseNQSNSQw3 j3ApF6Q/6MdUAOTLxzzwaGZiYUh0ubA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310279; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jBiuDcFai57ywCoGHCEF1oxPQjyCBa6ek76+fUhJDqY=; b=G9uy7rYTZH+avbvJQ7RhMYsrMvZnpe6JgDGWHTKjpuOpwAQYRBgijcVnPj96RU2XNXrZF3 zQX2il/bvXUF9lMsg0321gyYJaGpDmTdNhg0qtGmNJiPB3R/+iA5B5DNiy/Qw4BbCY+yb5 tbKLsTJ3VP/t2JxUC7FrC6Lp9nGarT8= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-252-kgGExcclMx2ItCIeuAt4Eg-1; Mon, 11 Dec 2023 10:57:57 -0500 X-MC-Unique: kgGExcclMx2ItCIeuAt4Eg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id F102F1C08BA2; Mon, 11 Dec 2023 15:57:56 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7E1691121306; Mon, 11 Dec 2023 15:57:54 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 24/39] kernel/events/uprobes: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 11 Dec 2023 16:56:37 +0100 Message-ID: <20231211155652.131054-25-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: 0DDB240020 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: y4semquceqfhaypq5z5ib5i89wqsdza1 X-HE-Tag: 1702310279-66176 X-HE-Meta: U2FsdGVkX1/+H7GFr7HE8cHWuduPOjPgRNJQ1cYRyc95RjaIfFBA+SPtWeNdWinEX3Ewm7XXtaJjfvD1sIor6l+tHZb48wCt1m/utCQEgqJx8lW6tNlxQBFlGW7UpldkVOple4+5vB3CRADL0h2Tt9zzTgogwEZyRhJZGHyhZkVPaEsMzbgjewviIObGtxLGnqqm+QPy91n7N68XOmFiluh7y4HdQYqYMqYhl8jKxq0mWkXIYSxXhEk5AIFcgK7bcCYwY93Y2J10vcYWHIvWzFSLrAylVFhCipK4jGRZGhBCIehPkCoyEpfhN3Sprve71a8eAD5gUWxKI76jK8/mIIZCJzskrdWK2TXdXdhcImtBlLmEAQwl0fettN00mruAp5kBZHsUcBKD3oCt8KAnUUGLmO5eyYinmUVBOlTrNx8/AyF3lxYOaP5rkAiHZCIFebw7kPyaFSMcfVAnAlkELv5yZU67sU5/x5RVrw2lnjlUtFPzFMO9UU8XwFLHR7WMjjuwcjTJV0/XzgfASjSn6EOsdGGDpmkJI2yg5n49e0Ai+54l38sPd5YDzvAzzu6XrIPQcxWbO6kukUBDRtxqber65r3wCMyV4yG+d9XYMiMcfvu0PdJVxJfA+SfQ7iZC/IkRRmjxc6iUT1YkhQjCbSDiwlIF324fjhC/58Yu+xpjXoPOgXtnQe8OQd4GCp+mm+aViwGWSSHjR7cyNykr9LQC8D2tQTNmq+fb1oDw11PzJ+B8kNrqv/v0E088t0W4b084xMqoBAoPPAwwID7MzZvwuIbtZYbMCsRtKo0SzfH+oXnHbwY5LTOYjpjY2z9Rguuu5wrYqQYsI4O2udJlfPD94QFRRqqufUOAVl5iIjLx4gwrL7Wfytl3v97LhWYHMtRL7sGjL3B5m1Q4kHY3LXrlYPozMC+xeu3ff0FMLgVlReRhr2OLs/rBntRTBlsAbO6sJ/e+ODFOY3tAc+P eQdqFE3m dbBOtHlfTrdStETgGVI5Gg5PuOobbVGbWonyWKaH6DgsoemcAFRWfiaJd9YTjjpOzt/I0wBPM0sa55Ckt+dgDumanLc4VQGEUGR8KMRFzcuk+XKPpfm6lJLkuHPcka1slBJ2avmg0BZixHf2JpvE4pOsjBaBVPPvoEsZ59wd72HoXO5TycMyuNv5PTrqrLsmspoAB7xZIhozuDSyuF9MwKlT8VA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert __replace_page(). Signed-off-by: David Hildenbrand --- kernel/events/uprobes.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/events/uprobes.c b/kernel/events/uprobes.c index 435aac1d8c27..16731d240e16 100644 --- a/kernel/events/uprobes.c +++ b/kernel/events/uprobes.c @@ -198,7 +198,7 @@ static int __replace_page(struct vm_area_struct *vma, unsigned long addr, set_pte_at_notify(mm, addr, pvmw.pte, mk_pte(new_page, vma->vm_page_prot)); - page_remove_rmap(old_page, vma, false); + folio_remove_rmap_pte(old_folio, old_page, vma); if (!folio_mapped(old_folio)) folio_free_swap(old_folio); page_vma_mapped_walk_done(&pvmw); From patchwork Mon Dec 11 15:56:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487464 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E4D9C4167B for ; Mon, 11 Dec 2023 15:58:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0D6E26B0126; Mon, 11 Dec 2023 10:58:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0873E6B0127; Mon, 11 Dec 2023 10:58:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF3FE6B0129; Mon, 11 Dec 2023 10:58:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id CAEC36B0126 for ; Mon, 11 Dec 2023 10:58:08 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 85F86806E4 for ; Mon, 11 Dec 2023 15:58:08 +0000 (UTC) X-FDA: 81554993856.03.343A5C7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id BF65840024 for ; Mon, 11 Dec 2023 15:58:06 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JZdd9qCZ; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310286; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3iv72ytjW+obJugegMCmaB6S7PQkIGz8PWP7EAJvF9c=; b=Fzzt8Sjkj1W/VnzSeyQvFJF06stHoXnJdJM8iF7z1/EaagsDrvtFcGXIIqTCarwbdPcgsQ d/5vc95sYTdkPmbY86wf/LKfKOSXZ5nDmMHiLm2xvA8a/UJaxgZ+OGC3FyocDKGGf/zrP5 itLXvdX5Pa1D1LSfu9ZWVxch9GD4DLo= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=JZdd9qCZ; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310286; a=rsa-sha256; cv=none; b=rmNo4Tz/flHCmcTMh48OkkLOhMknhGctseOMg86Hob4T8fGnpOIOZTGg4wON4uDqocAoL/ xDKaQrkfJljmYEC//QQy3wBotTH3dm8cEuA0wkymqzGZqS9Y3Akm7ZZU4+sK+jwcaNxkbB BlOLDzdd39wrKewbzrIAVzpYdnv0gP4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310286; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3iv72ytjW+obJugegMCmaB6S7PQkIGz8PWP7EAJvF9c=; b=JZdd9qCZaXUcdteSpzroijoPN97aPlgWZv0OGm5O40CE7aLc/lxMfGLj+kQzGfWOsuqgoA Uld6C/MTwcNVS2cNMu1T0k9KWG6z2U43f7qbrKhkAWYopV6ATZMlHPkXsejIx3MmIwnLWX lSa5oh01GVmbw8JpOvp6pTy62kLsz9c= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-328-jn5rk51mP9GOulr35P-yuw-1; Mon, 11 Dec 2023 10:58:00 -0500 X-MC-Unique: jn5rk51mP9GOulr35P-yuw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id A2A9610133AE; Mon, 11 Dec 2023 15:57:59 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 35F591121306; Mon, 11 Dec 2023 15:57:57 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 25/39] mm/huge_memory: page_remove_rmap() -> folio_remove_rmap_pmd() Date: Mon, 11 Dec 2023 16:56:38 +0100 Message-ID: <20231211155652.131054-26-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: BF65840024 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: t46ab9rad69c5a6j66yaq7zcx6tnnsqx X-HE-Tag: 1702310286-439403 X-HE-Meta: U2FsdGVkX18nuusOOr84+ZlMyPBNyqvLxOBaa00ZfP44OG5PSOkBEZgskLQEVfM4McEVqagq7tN7Zm1ZuhfmoimtIJCnMzs6XJC3GF4zed2ttsOVvpGXMhrx2KOWNO4tv8ilSdnk+0DJVreEFeBvbXgMV344Ndc3vzb/20uYBHLh9c2JruxGn1hdd1hZ39n5DVXoZfoqhBqe8OIiJkXEPAykNlyktdJkrstcntma2UJKMBjkjn3HOQa2+q9hICyoM4TXFJDDCAwaTV8G1x75U5SCAYGgcVoVrUtzlJmjnaLbvFf97htnpV8gOOUL2ORjbwELxJDxCuFTnXAHFcLKBSyEAKW2vghSHc0oJODhN8RxBRBEnlO2fczIw4agnh4sJkzS/3dlTvKovWglPpcWAqq4K7qyj5a+7u914asqFr//h6S+Ponl+ElKYdMthmHt/Po1qUGjv7e7sYNTsfUarWI6ndgyBk2s/bYzhXRoO9C5hwFalczFjoqPMf739hJc8V3oudJIvfFCQGJgcnOHZEt4OQyrWdTJzQTrAP6Lwl5lZ7Kg1s1ZgXxjMFj0HSqiElOl3RBPVp5OvyEQnGrrk4xjSULwxE2ss+PHGb6xImTdS9hDO9OPZnwA1fJlS1QBK/gruLs+VTW33iAIhRs3unmwikyjfkhHtEpfDpvkfj4t8wZd3mTIp8n8taxA+PX8RTQW7is5xp6I+FxpdSDaX4wVIrG+adixraZp/Q2c+1HjlNWzJn68Yy64mE7CtzvfpAyoNRRsz2Ru+vy4R+H2VyHnZi+NspMqwQckllngEierWc70IPAHtZBFr4dXTPFYWukZUK4EWkF9MvzjC4qhsID+U+riHiIW6wcuFE4snnm/tT/mcv8xPEBKRW+aFPwp5eqidzaS6b8a79h/mv9KwMm9Dnv5VOBQ3fM04ujEGZOJqJRV3vUeuOG5Yt7cpkhRm30ot/Z+7fkGh1zGnMH l2ZenWI4 xeUnFUKyeWqBhkkDltUSieANPQuwo4RLHn1G/Hn4+dx7sE41WeOng/9WkcpyVjr+gH3Xal6xBSBcVUskJDT70VHT4HbArd6Ood36u6/+Hhps8Jgm7CG9+JeYXE8YqNJ9h388Fab6+sP/S83SAPZwF6KohDVfmI8qg280mcsdvKLSks7c57y2fOUPoYJj1siqPqgvj3tlY1Hzo1DK246CieKqLqQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert zap_huge_pmd() and set_pmd_migration_entry(). While at it, perform some more folio conversion. Signed-off-by: David Hildenbrand --- mm/huge_memory.c | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index b03374d1bb94..cfaa8b823015 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1898,7 +1898,7 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, if (pmd_present(orig_pmd)) { page = pmd_page(orig_pmd); - page_remove_rmap(page, vma, true); + folio_remove_rmap_pmd(page_folio(page), page, vma); VM_BUG_ON_PAGE(page_mapcount(page) < 0, page); VM_BUG_ON_PAGE(!PageHead(page), page); } else if (thp_migration_supported()) { @@ -2433,12 +2433,13 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, page = pfn_swap_entry_to_page(entry); } else { page = pmd_page(old_pmd); - if (!PageDirty(page) && pmd_dirty(old_pmd)) - set_page_dirty(page); - if (!PageReferenced(page) && pmd_young(old_pmd)) - SetPageReferenced(page); - page_remove_rmap(page, vma, true); - put_page(page); + folio = page_folio(page); + if (!folio_test_dirty(folio) && pmd_dirty(old_pmd)) + folio_set_dirty(folio); + if (!folio_test_referenced(folio) && pmd_young(old_pmd)) + folio_set_referenced(folio); + folio_remove_rmap_pmd(folio, page, vma); + folio_put(folio); } add_mm_counter(mm, mm_counter_file(page), -HPAGE_PMD_NR); return; @@ -2593,7 +2594,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, pte_unmap(pte - 1); if (!pmd_migration) - page_remove_rmap(page, vma, true); + folio_remove_rmap_pmd(folio, page, vma); if (freeze) put_page(page); @@ -3536,6 +3537,7 @@ late_initcall(split_huge_pages_debugfs); int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, struct page *page) { + struct folio *folio = page_folio(page); struct vm_area_struct *vma = pvmw->vma; struct mm_struct *mm = vma->vm_mm; unsigned long address = pvmw->address; @@ -3551,14 +3553,14 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, pmdval = pmdp_invalidate(vma, address, pvmw->pmd); /* See page_try_share_anon_rmap(): invalidate PMD first. */ - anon_exclusive = PageAnon(page) && PageAnonExclusive(page); + anon_exclusive = folio_test_anon(folio) && PageAnonExclusive(page); if (anon_exclusive && page_try_share_anon_rmap(page)) { set_pmd_at(mm, address, pvmw->pmd, pmdval); return -EBUSY; } if (pmd_dirty(pmdval)) - set_page_dirty(page); + folio_set_dirty(folio); if (pmd_write(pmdval)) entry = make_writable_migration_entry(page_to_pfn(page)); else if (anon_exclusive) @@ -3575,8 +3577,8 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, if (pmd_uffd_wp(pmdval)) pmdswp = pmd_swp_mkuffd_wp(pmdswp); set_pmd_at(mm, address, pvmw->pmd, pmdswp); - page_remove_rmap(page, vma, true); - put_page(page); + folio_remove_rmap_pmd(folio, page, vma); + folio_put(folio); trace_set_migration_pmd(address, pmd_val(pmdswp)); return 0; From patchwork Mon Dec 11 15:56:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487465 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C81AC10F05 for ; Mon, 11 Dec 2023 15:58:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A74AD6B0127; Mon, 11 Dec 2023 10:58:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A25E86B012A; Mon, 11 Dec 2023 10:58:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8015E6B012B; Mon, 11 Dec 2023 10:58:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 6B3586B0127 for ; Mon, 11 Dec 2023 10:58:11 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 3C7A61C0F75 for ; Mon, 11 Dec 2023 15:58:11 +0000 (UTC) X-FDA: 81554993982.05.75F8AA9 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 7D8451C0025 for ; Mon, 11 Dec 2023 15:58:09 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gXhLSx8w; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf21.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310289; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VER2ukZQXsfAHa/RpU4plqm2RzyVwEC40zuZNC17KhY=; b=Xz+FW9moqk9YbVMhUxdL+SHNUcp+43MhdyyRfzImZg43Wyj9D6IxQvfJz8xCx91oNnFFF7 Kl31MEyzTZiW0xwSajnZ/3jBPW6aMbwktjSGxZzfwa+lziFdHEgD9PeNiOddsCZbwl5oSV Uw6d0LZeH6A6EYCEvU3fjwyPi8+ExQ8= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gXhLSx8w; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf21.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310289; a=rsa-sha256; cv=none; b=lEzyaVKEDW7SJBnIZf7SHpQj0pnZ4ECDnzFZ4E5o46my435T6T+BUqxpv7VFQ35F2cmncX lCPPpFh6fT71HSsd5NhZ+wxKq/Myph0n3idztVJ+fDR+9uk9IoyUG49w83DiJieF0ytqXW tHzFEngIEDAUkVmZeNSWzQdIEBKG7Vs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310288; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=VER2ukZQXsfAHa/RpU4plqm2RzyVwEC40zuZNC17KhY=; b=gXhLSx8wfWrx+SsqvBXh/5A6QnWb3MraItxEdms0AYXbBmLOn1F7PP4FbeiaWZO+HUp6tv uR6aQNCJiR6TP175pkE1MhfuTlR2u/3vQzmT+dpIarTC8P5b5CcwTeTzPxjmONpcyQc0nb z/bwRemW2NiHpt+wwtL53pr3CxdVAFE= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-554-O8xKFmHWPGuFDgNfrHONmg-1; Mon, 11 Dec 2023 10:58:04 -0500 X-MC-Unique: O8xKFmHWPGuFDgNfrHONmg-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 6E3723C28642; Mon, 11 Dec 2023 15:58:03 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 094C11121312; Mon, 11 Dec 2023 15:57:59 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 26/39] mm/khugepaged: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 11 Dec 2023 16:56:39 +0100 Message-ID: <20231211155652.131054-27-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: 7D8451C0025 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: muq6wfdp7wciryrswgpfynngru14ryj7 X-HE-Tag: 1702310289-238543 X-HE-Meta: U2FsdGVkX1+OH210Zd0EaftnVyCyRQ6KEPtgmQAdoh8Mi3+dqI+P2/V+iE2u5nvhjAK/9xNLBmHeGF28YvbwOWYBfqcgfQoCkQx+gzDqI+fjwRh+adjL6MKdvtN7zphmz11pxixhut7vjFpeiJNzqxHh/1dnC4u5p7OUr1RQvbNiK4LwRKKuSt+40VJXNIqW2HsG4dShxuOsxzTWtSq8nSFpF2aA6z0ITvRew+6HSobxVcafCnV2IGfb1wBs6EsfYhmB+jtxOViN7rHRiLrSt2d37FJCq5px1blotd9SHQKTdz1Xxf+tWJbcXe9nbvNiT2bytybnA/ireXEbzImnhdbhaCHuqtOOKW+5L/kKsHOVYjhZbTMzoh6aJZVtBHOdfgAtJJooVqCjirwhj19TH87gok9EQ+lQY7qpGMtbNozEy94zHu3Z3NB0KMJyOFg87mKrKCjsoQGPuHvHoDenvD963zWBJOdtd7rgo9S0I/5j9G3SGVshxCL71M/gD7YapdswY4rcmTf8gbYHD/16XtMKhDwdMveUpOx9fnmD+UmPudteIYhlOL4jjPcEHfmUvb++/WT2vx9/NDyhObl5ntJ2PvG8E+DybuFM8NPXwk474hAxn/oC1W1c74OPZTywTVxvf5/AljxOfJ4uCEWXV66JWoKNIvqxJ4FADS3YUm5auAhtf31MjwLe4sJ0UimuR/fiOfKM/Lj43fC/F4UwDr2dk8S7qoB1ELmZS3KLIQ/bIDFqpJRuZdhDqO5rr5DAfLH8rgkMawncMYsF6h8HMjkPKqR1Ib5ZSGlnJZM5v+zcR94W2H6LWlz6z3ifoK+ttF+x/oekljN/A7SMfchfOiq2CyhWwT+ynPzN1P0q1tUB9Q+cdv2YJu+POkEN0i4JqRHU1e7niLezhtoHej8QDRq1NJPXA20kwTlj6+/hkXyPEP6elrzzhrlYh5flFyCkAi2QnH1LRXp6sVERFUC scUXhjxp bAqvv6OTIeMEEvTUCm44NvQ2qi5znyPh1Yo7oiT1b/tYtnoUoOFV5M1i7mb4i9v/7FAVkt1KT5tNyPGEDkA9ev+abo1ysrdECC6+3clkeMRBZvuWs9NbbFjMJWuzNRQw9I+slmWdu+sVisbgag9DHrGwwp3r0AaDuGsVhKjOzC8NqltEBqUiQL7BMR6fyny5e/q7tHVcabDSSYgPBTfwmSddVtw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert __collapse_huge_page_copy_succeeded() and collapse_pte_mapped_thp(). While at it, perform some more folio conversion in __collapse_huge_page_copy_succeeded(). We can get rid of release_pte_page(). Signed-off-by: David Hildenbrand --- mm/khugepaged.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index de174d049e71..4d90c9548ec9 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -494,11 +494,6 @@ static void release_pte_folio(struct folio *folio) folio_putback_lru(folio); } -static void release_pte_page(struct page *page) -{ - release_pte_folio(page_folio(page)); -} - static void release_pte_pages(pte_t *pte, pte_t *_pte, struct list_head *compound_pagelist) { @@ -687,6 +682,7 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, spinlock_t *ptl, struct list_head *compound_pagelist) { + struct folio *src_folio; struct page *src_page; struct page *tmp; pte_t *_pte; @@ -708,16 +704,17 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, } } else { src_page = pte_page(pteval); - if (!PageCompound(src_page)) - release_pte_page(src_page); + src_folio = page_folio(src_page); + if (!folio_test_large(src_folio)) + release_pte_folio(src_folio); /* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats - * inside page_remove_rmap(). + * inside folio_remove_rmap_pte(). */ spin_lock(ptl); ptep_clear(vma->vm_mm, address, _pte); - page_remove_rmap(src_page, vma, false); + folio_remove_rmap_pte(src_folio, src_page, vma); spin_unlock(ptl); free_page_and_swap_cache(src_page); } @@ -1624,7 +1621,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, * PTE dirty? Shmem page is already dirty; file is read-only. */ ptep_clear(mm, addr, pte); - page_remove_rmap(page, vma, false); + folio_remove_rmap_pte(folio, page, vma); nr_ptes++; } From patchwork Mon Dec 11 15:56:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487466 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D390CC4167B for ; Mon, 11 Dec 2023 15:58:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DCF9D6B012A; Mon, 11 Dec 2023 10:58:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D80A86B012C; Mon, 11 Dec 2023 10:58:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD1756B012D; Mon, 11 Dec 2023 10:58:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A115C6B012A for ; Mon, 11 Dec 2023 10:58:12 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 77B2E120466 for ; Mon, 11 Dec 2023 15:58:12 +0000 (UTC) X-FDA: 81554994024.20.09FD23E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id B204A1C000A for ; Mon, 11 Dec 2023 15:58:10 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TMVEKLFK; spf=pass (imf20.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310290; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QtgFnZdp8Ufq96QYGYH/M8uGtDulU5aA6agoBTCHJ/U=; b=MVm7dnADQ6tezGZEVcVwlZBBYx6xdSfjPhLBC+5KX3GYMPM3hmMlCs5xDyV8xhQNy8sO6Q J+ulA3Ibj6KKR7Z6fNvVc9xIahr2XSbvXpOx1fkAc+EOKk6UhQIsphefGU8O+QRg5hbbQ2 jAvr673BGwTJoXtA3tURrNKAH8e48dg= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TMVEKLFK; spf=pass (imf20.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310290; a=rsa-sha256; cv=none; b=Fo12AKeyXhO2iwwgJXDE5jAYSinUVJq+F6pXfBvWosSivCml53OBlyGCpRhtS4rW8PN8q+ YNWXL5HZ551vugLvgb1csPtICAwUaaiydxuEx205TCqMWYpxyruPAq0/0M3AidYDTep/a3 vhEJInkTlty7vMVmBMORKOZSwxatdbM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310289; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QtgFnZdp8Ufq96QYGYH/M8uGtDulU5aA6agoBTCHJ/U=; b=TMVEKLFKA43o5hs90V6x6wjP9YIybRoVe50r8+33MlRr2A1cME23YVbaPfWXWBwnDx8k80 BOciUwhaEmt+OLccOQ6Nf9jsI5wDBOeTgQJ4ky/Q/PHWQ00TqAILQ8HyUlHDQRqf6vTMNn MBq34SscSGetf19+4C6YeHsQVCP1DpM= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-578-Q5gG0FfpMgG6qcIyHy9Eng-1; Mon, 11 Dec 2023 10:58:06 -0500 X-MC-Unique: Q5gG0FfpMgG6qcIyHy9Eng-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 763472806408; Mon, 11 Dec 2023 15:58:05 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id A56501121306; Mon, 11 Dec 2023 15:58:03 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 27/39] mm/ksm: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 11 Dec 2023 16:56:40 +0100 Message-ID: <20231211155652.131054-28-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: B204A1C000A X-Rspam-User: X-Stat-Signature: z17fk9ykqpf8j77rpkiaoxbyyt93nyhm X-Rspamd-Server: rspam01 X-HE-Tag: 1702310290-233267 X-HE-Meta: U2FsdGVkX1/NKh4L9P+X7fDZz72lKcA74Zsx+J5icjvBCXx64DMiYM/+JX9er2pEpxXw5pqHSoqusll4Y3Abzfnn6sXraPBbwHKe3qHPBPzj3/Tzol709fsEHcPB5RKu+R8le6N+rnuhE5yTUIN4TDeJNEf/O+g4NyVPY+Pq4X/OgBYCd30lP0YRk6mQjls8LsUxzxFNP/1lMlhmdutw5aU8oFnK1YzyWpBBNf6zE4aCG7lFEB+4At1dxcadnoyhu0ji8ji18D7vRTDy8Jxgbp78f3nMB8Y7cLjaPkHvd+VRWL3tHnfxE27uZOsSRhKvkrKoOtBESTGX/dPrXr73a9KXhHfDvl62MZ10EzIoSMnmHkrxQVGU2t3WfgzxGnOEMuZKMoYfzhsq9VpYkiXvUnaUxrI5amrD9xZBa9cQt/WjPrwPmKy+QzSwG0/L+whaTxLrBD7G1LveEreRMIh72rMRai24TDBNhR+GeiDdrRzisV7uS88Mmf/EVcGF+IhfYU1Rf3NG0e6Ow7001XxOu4KaAXvYb5vifXyq1fRiVpm+t9423s8tSzZkAZHlIN2wmFwxtMzGZyNpH5Pk/M0H+gyx6zUocoRopqUNtUViAl7oDeKQZApOExWTzkbQI4Im/GCL8qaibjTgrldMHRXHNPN2Sz3Ktum+WMfI86h4Oqus2tXyN70nxjEVHKIZnAPpaeaeABda7cE0Gw9nZcX2NzCrut4ps13ulzYlpQJu8ZZbKfKRmGO4B/Jz4mx7V4nFyjS3US0xX3G34WeDkYEfc6t7QfMcel79y5HhzBvyQxRLN22V7j+pWZa9C3SXjfrkHKs96XoDVhCYhvyegn7brqJIOW/xGetcwRb8yrfTnUZ/VgfGwD24xMg/rR83ePKfERGVY4DKoCnzYXAm6FyNqVUw9L+YybhbROZNWrLIWBaKOEuIjYOrJJVQ946MSu4vo8q3Hs940l4lqnQYCFG 9IOnXHkW 8/3OxE3AeQ/MHAS2mDF5u7GXokfz8Kau6dYEOiii0yKyhDF9szQFO69aG41Q3sGQ8J630dEer5DolXNBz2Pf41A/0r4UFC1Y6rpt/+N/C2suNgr+PsfNMwKXvdiRhaQ4gBHOxwv1vSQCMGTIdzKrxSvKYTsu0m2NatrGXafmfjKaXvSDVs942iX4o/9ZUgSdqsLyuiNER1pr4D0t/TYQtIi1LPQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert replace_page(). Signed-off-by: David Hildenbrand --- mm/ksm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/ksm.c b/mm/ksm.c index 2b6888ad1470..b3d0cfaa2533 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1279,7 +1279,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, set_pte_at_notify(mm, addr, ptep, newpte); folio = page_folio(page); - page_remove_rmap(page, vma, false); + folio_remove_rmap_pte(folio, page, vma); if (!folio_mapped(folio)) folio_free_swap(folio); folio_put(folio); From patchwork Mon Dec 11 15:56:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487468 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0226C4167B for ; Mon, 11 Dec 2023 15:58:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C4A9A6B012E; Mon, 11 Dec 2023 10:58:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BF8C36B0130; Mon, 11 Dec 2023 10:58:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A70FF6B0131; Mon, 11 Dec 2023 10:58:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 914936B012E for ; Mon, 11 Dec 2023 10:58:17 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 5FEF7A1B33 for ; Mon, 11 Dec 2023 15:58:17 +0000 (UTC) X-FDA: 81554994234.08.2694EA8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf28.hostedemail.com (Postfix) with ESMTP id 82DE8C0013 for ; Mon, 11 Dec 2023 15:58:15 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=d4Vb+z6d; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf28.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310295; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=e8cPKSG16Q/IRVzSDalc66DZf+rAdIx9s1sCKVbyMk0=; b=1znz12cWNtVxg1S4kkNuQ2PNxFbEkVIdMtj6b+7/oh5jl3YzF4dkhpFslXSb2Kl3QmBRA6 SyVEprxwzuNh6Od836qTEIX3o4BpPY3G8lgoCRGva2Cg1WyJi5bfTaY7VwwtLSqVYcQJL5 ivbXmnXY3DZQ7Dy049f2cIR7GRlCMOQ= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=d4Vb+z6d; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf28.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310295; a=rsa-sha256; cv=none; b=tVvZ//RTGL+O/r0mnNy1X/7ZnnffEmXJW2FvdWKoIJ53OohU4nxFFaID5PLz4+YfSpjKyv 3EvetnwKLK+p70ApT5FeQSkhm2HuBNjE4h3+P4MoEgbpItCDz95B5/g6GzRdGAnlWnmlux rdAA54MDokNGLHA6+sV5zkygCrJ7u6E= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310294; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=e8cPKSG16Q/IRVzSDalc66DZf+rAdIx9s1sCKVbyMk0=; b=d4Vb+z6dCUNtA/g/7M5q2miDz98eu+Emq3F8PMSDAlJKspnDiT29+eWR2G72b5YrAE2enE Y4A82L/PXYD3Ev2HeUhtsVqSiw/v14yijxn4VgOcncey4diVa21WYLD1hFDywX8kA5UcKd cfdTlDof24zUGGyyXRwS4qgGxjNGqDE= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-611-Qc9Fji3rOuid9DMrJtMfFQ-1; Mon, 11 Dec 2023 10:58:08 -0500 X-MC-Unique: Qc9Fji3rOuid9DMrJtMfFQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id CA5991C08BA2; Mon, 11 Dec 2023 15:58:07 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id BCEA41121306; Mon, 11 Dec 2023 15:58:05 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 28/39] mm/memory: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 11 Dec 2023 16:56:41 +0100 Message-ID: <20231211155652.131054-29-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: 82DE8C0013 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: ttak5y9wicoihsyg3zx1e5bqcikmp5wn X-HE-Tag: 1702310295-818055 X-HE-Meta: U2FsdGVkX1+wJY6jGX7lIGCmWqH/7l72LKbrirHLrxZVY2TFejLQRF+f20yvHkpBrOahP0SL9Uq/Kr6O/NmL0UsUeit9YgZ9e87mHhFlo7QGah6xMjngQ91YfXCj9wV0LcFJC8LMBvz4rrD+v2JdeVfXAIIkQDOxVyu1iamnctuzVxAKWGkJDkl3xF9LDEWiJwmyi++2CI9igb7+4jSD0xqjwbB6tu32yHu0d4rJuQKTukmqbLbHKQHbPWJVyW+lcOgKudZLqtwai3+Q4ZqtUzTm3d6CehSTpjMGAVVczqNnCfFa09+DkSUz7i+N+vvDNRE03LgojUUV/6Wegf2jMtI091M9LGemnafg4/5ZXeK4PrNT1yh4fxt6roSgAOctcabxr/1KSJj1B4LijgOySfOojYOcZVgYJCQhkLRGADdwQqOfY/P2T99fpOAwCiHo5FLtYzzpRlKVtLfvudeLKevz41c+sObZjxbea+HAULv3SPJMEWHUqSvze7R3nZ+E4t1i2i5knjpk6FCmEsw1EX18/7jEillzXjUaaCEfyOCmELlEfr1S5UmsKl64C6lmTIS25bPTzogFm3JFR5WHZ3HRKtbRyNoNUH6h4A7SVJEMJK3Qvab3heCKK8aVeGozJNgqnQsHZCB7VlyNZRFJRaT2D2nEA08A9BXWDSY4/czbYvtnNPGSnzS8SKMsfWLYpHb49/pntT4Uku9wv+FQi/zQYKnlJpTEj3ca4bk2jpDTh+iMe8TxTJIr++vA9wO1gH+PfRBQ4rx9rPx/hSOQH8j8mM01to/sYc3ZGdx6D1aEtW9ILNScScxBw6ASKf9mI/uq6ug2N6nEEWq7+Sndlw3P+3hoDe4gol4kxMsLnvUsQjly8m39fnzd5MKHNzXWtQzjZXvMx+nDQqsvMDiei3nO2gWCdNzPMS+e19FLvT4PEkq1bJp6eVMpBEUEZS3iIL5k8R9sZS6ybkyXy8b w6+6iYNF R5moRGxJkSNfNYioNgwGc7YOGXMv9Y7Hvawwis9uxOiduenx2bPLvjekGYL1ouRx3Yu5krp8dsaOl9eAvkDOWHqbmNN1A2WwfkOQuTfE0soM6H71XZR3VMQcDYkrxKI8Azk4Ffs8EHbBZfEyAOgdIN1fTh+vHxm4zORpheImP4BIOTMtBnNlQ0ehg47Tm/OZEDlkBSgEQIMS3rlQqHfGiuW1+5Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert zap_pte_range() and closely-related tlb_flush_rmap_batch(). While at it, perform some more folio conversion in zap_pte_range(). Signed-off-by: David Hildenbrand --- mm/memory.c | 23 +++++++++++++---------- mm/mmu_gather.c | 2 +- 2 files changed, 14 insertions(+), 11 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 97e064883992..9a5724cf895f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1434,6 +1434,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, arch_enter_lazy_mmu_mode(); do { pte_t ptent = ptep_get(pte); + struct folio *folio; struct page *page; if (pte_none(ptent)) @@ -1459,21 +1460,22 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, continue; } + folio = page_folio(page); delay_rmap = 0; - if (!PageAnon(page)) { + if (!folio_test_anon(folio)) { if (pte_dirty(ptent)) { - set_page_dirty(page); + folio_set_dirty(folio); if (tlb_delay_rmap(tlb)) { delay_rmap = 1; force_flush = 1; } } if (pte_young(ptent) && likely(vma_has_recency(vma))) - mark_page_accessed(page); + folio_mark_accessed(folio); } rss[mm_counter(page)]--; if (!delay_rmap) { - page_remove_rmap(page, vma, false); + folio_remove_rmap_pte(folio, page, vma); if (unlikely(page_mapcount(page) < 0)) print_bad_pte(vma, addr, ptent, page); } @@ -1489,6 +1491,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, if (is_device_private_entry(entry) || is_device_exclusive_entry(entry)) { page = pfn_swap_entry_to_page(entry); + folio = page_folio(page); if (unlikely(!should_zap_page(details, page))) continue; /* @@ -1500,8 +1503,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, WARN_ON_ONCE(!vma_is_anonymous(vma)); rss[mm_counter(page)]--; if (is_device_private_entry(entry)) - page_remove_rmap(page, vma, false); - put_page(page); + folio_remove_rmap_pte(folio, page, vma); + folio_put(folio); } else if (!non_swap_entry(entry)) { /* Genuine swap entry, hence a private anon page */ if (!should_zap_cows(details)) @@ -3220,10 +3223,10 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * threads. * * The critical issue is to order this - * page_remove_rmap with the ptp_clear_flush above. - * Those stores are ordered by (if nothing else,) + * folio_remove_rmap_pte() with the ptp_clear_flush + * above. Those stores are ordered by (if nothing else,) * the barrier present in the atomic_add_negative - * in page_remove_rmap. + * in folio_remove_rmap_pte(); * * Then the TLB flush in ptep_clear_flush ensures that * no process can access the old page before the @@ -3232,7 +3235,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * mapcount is visible. So transitively, TLBs to * old page will be flushed before it can be reused. */ - page_remove_rmap(vmf->page, vma, false); + folio_remove_rmap_pte(old_folio, vmf->page, vma); } /* Free the old page.. */ diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 4f559f4ddd21..604ddf08affe 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -55,7 +55,7 @@ static void tlb_flush_rmap_batch(struct mmu_gather_batch *batch, struct vm_area_ if (encoded_page_flags(enc)) { struct page *page = encoded_page_ptr(enc); - page_remove_rmap(page, vma, false); + folio_remove_rmap_pte(page_folio(page), page, vma); } } } From patchwork Mon Dec 11 15:56:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487467 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C114C4167B for ; Mon, 11 Dec 2023 15:58:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0D5546B012F; Mon, 11 Dec 2023 10:58:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 089D06B012E; Mon, 11 Dec 2023 10:58:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DCD3E6B012F; Mon, 11 Dec 2023 10:58:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id C738F6B012C for ; Mon, 11 Dec 2023 10:58:16 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9E20DC0768 for ; Mon, 11 Dec 2023 15:58:16 +0000 (UTC) X-FDA: 81554994192.12.97FA852 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf27.hostedemail.com (Postfix) with ESMTP id E1FF440018 for ; Mon, 11 Dec 2023 15:58:14 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=W6iXoffp; spf=pass (imf27.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310294; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zZtDmTKZIWvKon5vN+gkyoFpv7PryiRHfxxjTiLIEgc=; b=gmLD48wnpmT1iTYY8AvMXxmNVWVENSSVhMDevyu90xNc6GFuevMprWLplg/iuYaTLtqIwg 9K6SpeP0gLYx4kbYWC+u3eWax3PZ2ZJPQ8nVX5XHA6lHptbsDDPSSB4ZJ9HLkFwAI54kWN Yd385uW5/CFChwmkP9bf6n9h/ZcCtmY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310294; a=rsa-sha256; cv=none; b=uH3SyfI0XX3GMYMnz3ZFPuXe4U1aUDvUhlltFGWeqvYDxxXQhnyi/HHFF4wmglhodxymNU OI/EHovuEIDolA78KjsFlRhtGPmeFsWGz8hydMAHuD2ZVNm67M1mHUIDfNUJVyOh8j2mYY kDqIMhsfJ1W+2GtqDdKCBnqzxOU+tDA= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=W6iXoffp; spf=pass (imf27.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310294; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zZtDmTKZIWvKon5vN+gkyoFpv7PryiRHfxxjTiLIEgc=; b=W6iXoffpl2Bdn1QFsLKEPvcM/O9BUjFHaEqyFjRZswdhErgE12B8QdwmP+eeqxVN99udGA izQaBH52FE77PrWhluTGRYYtQOIkDBrqGqW9GYFDfYNAigyju2WOg+PkWwPcJYdf+/gNLP umSetY5FM7fVoKiVzweY0oGJWxiIuE4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-617-k9oBbGedOyagcPkWkippwQ-1; Mon, 11 Dec 2023 10:58:11 -0500 X-MC-Unique: k9oBbGedOyagcPkWkippwQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 7A20A870835; Mon, 11 Dec 2023 15:58:10 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 1F6FB1121306; Mon, 11 Dec 2023 15:58:07 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 29/39] mm/migrate_device: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 11 Dec 2023 16:56:42 +0100 Message-ID: <20231211155652.131054-30-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: E1FF440018 X-Rspam-User: X-Stat-Signature: 3yh4pdmykyupini6knmaep59sx5up4nu X-Rspamd-Server: rspam03 X-HE-Tag: 1702310294-465814 X-HE-Meta: U2FsdGVkX18G5NFnIo/OZna6fKGSP7DGGTRu2nhzse64G4MNeLHBtt7G7wh0AtvSnToRQ0ODY+ok0OiFx0HYf19iYp6FDTFYX9bbjTPEbodsWm8l79D4wkSc5mUqcyLEn5oiBZnRBsVd1fpdlFmg9P4qeO64iLmYVM5CP488oZ7DPPLFR8H4mWoydqZp2//zmUF1uDyvGm8Ln/Yy/3wydKbdNmZe+LP2pEzgackbuudrHgH27ULi4YdycffNimUB1jE8OpL7Ki4iUqqrOTMtWbHJD6HsTa//mwyU9LEErr3O8GAsRlnJKWfVMheocEaQ+JH3MqHOEz3DeE/CXo2bwIy0jjL5B5rNgwOtX3AgwouxibEEvsYZgH73MG7NQsvn2FtCeczbb4iPJQa+cnTtdCPmgCo63lRBtrvbt7ZmuGew8fbAtePtV57//azeM5olh/BHClsm6lD+wbIjrcCuT7ofvwjn+h9K85UTx2OF1md6tZyzwwc0cg8mm73cOsv4VJ5RKn1jlB243x9g+Qa/E0eLbE8kaJqT+q5/wHW2LpfJ8N2MptyVIYFDCLV0wo1+70uaKxDrxVlP3Z2D0j4tnsMF2+kaOmetFeR+BfVZnz3fRPcK6win7dZaamAiHdDvBh793DhieTB9FlU6JE3tguZ2rYs1VrC0UHeMOOII1819eBzUujrfnLzSke6IaCBAvPcxHylD+IQR2Ilx5ePqIUhCatYIatQ/q+lkaLBblqG3wqh5WPnPOWoAO3fdS6XkYgbPaRRkRDYLn2/+PpEisNE9PlNEyLJpUagqwsbb1BfDxGw4vlWFTyTgBgUE19HlPiQTrkylhQV3LjesG0PTHH0RgndqtVqRC1xpOoBkXtFQqXVnf8TSzzTTPT4ob+ZfqoS8FxK/AP0LPtU8/wuhq3qjUeF0Bn+2RvhcudURuUTivk4UV4sNwD1E5UZJOS3G+KlIS9s18BgGOeMNcES HnKnsdl0 OR1NKdN/221P7w/m6jp79J/vJRZeU9xbf2YPdBVCZjCwmzmTKBoF2MAgL0WTc0J/yt93EFSqThIG85OXRfnXqrw/z7X9+/T/PbzDVMX36GtgTANGYLBGQq5hkf+riXX0BNVs2+RTQZFxNVSil2gtJas/nugqHfVU/G5bDB1khh/oSJxA2fwxULrgMIyyr/oxKHRi1nmqS0snUI0IfE1y/2+p+mQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert migrate_vma_collect_pmd(). While at it, perform more folio conversion. Signed-off-by: David Hildenbrand --- mm/migrate_device.c | 39 +++++++++++++++++++++------------------ 1 file changed, 21 insertions(+), 18 deletions(-) diff --git a/mm/migrate_device.c b/mm/migrate_device.c index 8ac1f79f754a..c51c99151ebb 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -107,6 +107,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, for (; addr < end; addr += PAGE_SIZE, ptep++) { unsigned long mpfn = 0, pfn; + struct folio *folio; struct page *page; swp_entry_t entry; pte_t pte; @@ -168,41 +169,43 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, } /* - * By getting a reference on the page we pin it and that blocks + * By getting a reference on the folio we pin it and that blocks * any kind of migration. Side effect is that it "freezes" the * pte. * - * We drop this reference after isolating the page from the lru - * for non device page (device page are not on the lru and thus + * We drop this reference after isolating the folio from the lru + * for non device folio (device folio are not on the lru and thus * can't be dropped from it). */ - get_page(page); + folio = page_folio(page); + folio_get(folio); /* - * We rely on trylock_page() to avoid deadlock between + * We rely on folio_trylock() to avoid deadlock between * concurrent migrations where each is waiting on the others - * page lock. If we can't immediately lock the page we fail this + * folio lock. If we can't immediately lock the folio we fail this * migration as it is only best effort anyway. * - * If we can lock the page it's safe to set up a migration entry - * now. In the common case where the page is mapped once in a + * If we can lock the folio it's safe to set up a migration entry + * now. In the common case where the folio is mapped once in a * single process setting up the migration entry now is an * optimisation to avoid walking the rmap later with * try_to_migrate(). */ - if (trylock_page(page)) { + if (folio_trylock(folio)) { bool anon_exclusive; pte_t swp_pte; flush_cache_page(vma, addr, pte_pfn(pte)); - anon_exclusive = PageAnon(page) && PageAnonExclusive(page); + anon_exclusive = folio_test_anon(folio) && + PageAnonExclusive(page); if (anon_exclusive) { pte = ptep_clear_flush(vma, addr, ptep); if (page_try_share_anon_rmap(page)) { set_pte_at(mm, addr, ptep, pte); - unlock_page(page); - put_page(page); + folio_unlock(folio); + folio_put(folio); mpfn = 0; goto next; } @@ -214,7 +217,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, /* Set the dirty flag on the folio now the pte is gone. */ if (pte_dirty(pte)) - folio_mark_dirty(page_folio(page)); + folio_mark_dirty(folio); /* Setup special migration page table entry */ if (mpfn & MIGRATE_PFN_WRITE) @@ -248,16 +251,16 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, /* * This is like regular unmap: we remove the rmap and - * drop page refcount. Page won't be freed, as we took - * a reference just above. + * drop the folio refcount. The folio won't be freed, as + * we took a reference just above. */ - page_remove_rmap(page, vma, false); - put_page(page); + folio_remove_rmap_pte(folio, page, vma); + folio_put(folio); if (pte_present(pte)) unmapped++; } else { - put_page(page); + folio_put(folio); mpfn = 0; } From patchwork Mon Dec 11 15:56:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487469 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53498C4167B for ; Mon, 11 Dec 2023 15:58:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9D68E6B0130; Mon, 11 Dec 2023 10:58:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8C5046B0132; Mon, 11 Dec 2023 10:58:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 73CD26B0133; Mon, 11 Dec 2023 10:58:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6146B6B0130 for ; Mon, 11 Dec 2023 10:58:23 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 396B31206EC for ; Mon, 11 Dec 2023 15:58:23 +0000 (UTC) X-FDA: 81554994486.12.DE1A897 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf17.hostedemail.com (Postfix) with ESMTP id 56F4F40018 for ; Mon, 11 Dec 2023 15:58:21 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="hmRBuw3/"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf17.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310301; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=eWMSqygXIPXuXZ+kMnrkRT/FVjt8oLCVIg9Vpm7UnEo=; b=NnP+AlwcAdcgTRr1u/iYW0oQ+78zznx0s3s4UCqBwhQGI92OzRWFErRsGMpi6iG92AwGPY w5Vn8cN+fC7l0zo3FgLKw3uZHEwSKvursQIIi7sDybnzwk4hfhsV3r+DpU8EGg4F9brBWv f8pXVbEScTaUIk6rF2R9hIH7XtLbKeA= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="hmRBuw3/"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf17.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310301; a=rsa-sha256; cv=none; b=DQxAWVB7mkFJ8/bM8Sl0aqg7K67LGHOruR8UVm9Hk8mb87t7Ry2ylN3IgyYjIfjNoG5tAX F0VDQ3KN8OczoOk5C1jTEGrMdmVG5P3HEQk2YgAaz9aFis+Q5Hf7Hf1Byz9V/eCxhH2jn5 LSb3+edgxFygMaZ4mY63c2we8jQsH0U= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310300; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eWMSqygXIPXuXZ+kMnrkRT/FVjt8oLCVIg9Vpm7UnEo=; b=hmRBuw3/mKgmjZOa93/0QL927d79Wk+Ca1J7P3B8wFp4oj6K3naBEKUE4bfc+4+t+3M+V2 CDb6MCeQPlg7s/ls7sDLRzHU5IqfsNsdSLbiHpmNy4Lzd+AiboOQzeLUHbYCdgzY8kViOq AG4EZ7DMdTjHWRktIzE4GzFZkQ9jhXk= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-179-uhoPHFudP96VNqAzzD5Khw-1; Mon, 11 Dec 2023 10:58:13 -0500 X-MC-Unique: uhoPHFudP96VNqAzzD5Khw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 29BE038425AD; Mon, 11 Dec 2023 15:58:13 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id B38C31121306; Mon, 11 Dec 2023 15:58:10 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 30/39] mm/rmap: page_remove_rmap() -> folio_remove_rmap_pte() Date: Mon, 11 Dec 2023 16:56:43 +0100 Message-ID: <20231211155652.131054-31-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: 56F4F40018 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: rabhcrbuwqdn8mcaz4gezw1ky39xgqj9 X-HE-Tag: 1702310301-236879 X-HE-Meta: U2FsdGVkX19Ir3hXdjtcg6fFGvWSdeoPNCP2KALgYYkbpxoe48HlisaV2Y7BMultouKACc1K7OefM676RHr0TeejMkbsgL7Rl9Y/USSV2oufSI76ifQ+fren1sxFsEZ/OJN7P8Y6VlkcSIPJzjFPIi4vrFPkvHBu5bAoPDaEX9t1TA8fV7foozR+QqpjlbqqHw7rQwb1QjHAMOBBFEa1k44m7BkAxNAebM6NtInGnHfz2x4OpzR5+2EeaxMsWSJc4SDvztFMJVRX+bvfN3V1Nt8wOT93QcZ0P9F3PYPE01Q4ZdMxm/g591Wtm0JHqqZ843BMeDz/ZPTJsoAtZWvOfl2DD+qjCBOYTO9+9AsHNWhciQNfTV8CCA0n7PCIbP0E3UIv4ctAJ/wWhners6w/BtoTUOaLiAI+JM9gEQzCSJah3FJlIqVYTTb3Hpb9g4lSljOT+CVZTHHGg35RbHAPB1GUVsBc8u7S994f6z/h4G6UG3+0w5zCaTL8rgkexCV9rwCEDl1gO/faEZ2VKgxhi83oAUY1W5/WWqlpw0N6kVlMruxacn2WldoCZWnZpvzjHfWmwPPD5eBv1k9Z0yjccLMOL0yagylzSUTAYXmmhRoDC4c8e657sej9VcKh54jUvKMZKZ+5vyNW94PV8c1Gxs4V/WPfM2H7xhjmeOY56UqxN3XYAYWJyv2rJLuFXHvgoz8RvA6cRJ/Z9pg37tmLbzMvPqaoaxTSZASTG/Q3iZXQSptmqSjoual/e2IVsN9RxyNjNXrVoNA4l87V7rVMaDAaXQfy+onkzImebfzmaEJdsQ83kwp3X//E0MwIwfEsJftSEE94CkpCG9gQCVuJ8mTfHWNruPXzKjxaP6YyyByT6TiOkwMIBOlNaLDlVQ56cGfKsuoUmcZQnif90+HLtOTH92iYYZquTxDlWBe9Fpy3E4X+6PqqoHxfPVO853M5dsGGSRaFGYegknTVKkG o8IehQaV XgnyM3lOqTXi8Q01vQALj39uMYVUxXqw6y3TizkiQxHdmVHsl8T8dVHyS+87SyRshTPQvupZ5xuXL7DEPGs3UTSqVVVjd6rjS62QvSYpLL1/5PBpafKXGhJUm5xfpyf2b18+bEqr3/SWtn4Ir/RkO+/IonTjGuj0z1mAIb9FaQ25DUqV+Iinu6/+Q95AXhNqT73FFUBm6R+XRpibCiu/lIuCAr+g/HW+paDono5URzupfv6aBjU2sIKzOJRs0VVwEcrFiEHPscYBGRpo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert try_to_unmap_one() and try_to_migrate_one(). Signed-off-by: David Hildenbrand --- mm/rmap.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/mm/rmap.c b/mm/rmap.c index dc3be5807cee..233432f08e36 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1649,7 +1649,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, /* * When racing against e.g. zap_pte_range() on another cpu, - * in between its ptep_get_and_clear_full() and page_remove_rmap(), + * in between its ptep_get_and_clear_full() and folio_remove_rmap_*(), * try_to_unmap() may return before page_mapped() has become false, * if page table locking is skipped: use TTU_SYNC to wait for that. */ @@ -1930,7 +1930,7 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, if (unlikely(folio_test_hugetlb(folio))) hugetlb_remove_rmap(folio); else - page_remove_rmap(subpage, vma, false); + folio_remove_rmap_pte(folio, subpage, vma); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); @@ -1998,7 +1998,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, /* * When racing against e.g. zap_pte_range() on another cpu, - * in between its ptep_get_and_clear_full() and page_remove_rmap(), + * in between its ptep_get_and_clear_full() and folio_remove_rmap_*(), * try_to_migrate() may return before page_mapped() has become false, * if page table locking is skipped: use TTU_SYNC to wait for that. */ @@ -2291,7 +2291,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, if (unlikely(folio_test_hugetlb(folio))) hugetlb_remove_rmap(folio); else - page_remove_rmap(subpage, vma, false); + folio_remove_rmap_pte(folio, subpage, vma); if (vma->vm_flags & VM_LOCKED) mlock_drain_local(); folio_put(folio); @@ -2430,7 +2430,7 @@ static bool page_make_device_exclusive_one(struct folio *folio, * There is a reference on the page for the swap entry which has * been removed, so shouldn't take another. */ - page_remove_rmap(subpage, vma, false); + folio_remove_rmap_pte(folio, subpage, vma); } mmu_notifier_invalidate_range_end(&range); From patchwork Mon Dec 11 15:56:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487471 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E74FEC4167B for ; Mon, 11 Dec 2023 15:58:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 82E596B0135; Mon, 11 Dec 2023 10:58:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7E0686B0136; Mon, 11 Dec 2023 10:58:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6066B6B0137; Mon, 11 Dec 2023 10:58:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4CD0C6B0135 for ; Mon, 11 Dec 2023 10:58:26 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 24036C0757 for ; Mon, 11 Dec 2023 15:58:26 +0000 (UTC) X-FDA: 81554994612.01.EAFDE14 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf26.hostedemail.com (Postfix) with ESMTP id 5D6E314001C for ; Mon, 11 Dec 2023 15:58:24 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=A+xEM+zn; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310304; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=j7yPJXNweA3+MouJWc5Ro2+qKjmp/wEHVCvW0WWVjak=; b=C8rliuZ2LbtMnqLAnvoEdFGRQfMYTq+wfF88U+RJ37jA/pCyK8RKPFCWsLqG98RGbtdWJo Lh+vxWCFMGd4yRJ6G8mGNgSI5eT4ALkOSGzs0aF/gh7HRCXeSqRL3zP1QdSM0XDqOmwZ9H rCGMhuqjVvlBDCl2T6xtkx64f/9WNSo= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=A+xEM+zn; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf26.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310304; a=rsa-sha256; cv=none; b=gGkRGfs0fB4yDOWisLQM7ywv9WlNtw2jPCwI8nA9oEPUOXNn6mJCYUUdnN4BnUa7T/n004 0MiviHgj1DLbI0itgrqDgQVxo2gIFn8SNwuYceF/9OMh6XLxdRlYcboMncVkBu5K2V4Yzw iD78MLyl0KncPVp/tYpubWt8f1pDJYA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310303; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j7yPJXNweA3+MouJWc5Ro2+qKjmp/wEHVCvW0WWVjak=; b=A+xEM+znlIJIEusdJBZKudn48CMQkozHoy2tXYbfcGXHyS6RG/+jSdqSZPuVyDoDwm3Zch ya22msoICM4rJk+TTpwsJYkDMtyLkCHj5UpeXCL5Rrf86YZo9wTlR9C9sRSvrJRRG3I9gM nYDdOtAjlYxIdYPUzOVaNoig9UNJNhQ= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-312-4EKUpxx8O7a6a8BNS2xAZA-1; Mon, 11 Dec 2023 10:58:17 -0500 X-MC-Unique: 4EKUpxx8O7a6a8BNS2xAZA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 07746848A6F; Mon, 11 Dec 2023 15:58:16 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 87CDF1121306; Mon, 11 Dec 2023 15:58:13 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 31/39] Documentation: stop referring to page_remove_rmap() Date: Mon, 11 Dec 2023 16:56:44 +0100 Message-ID: <20231211155652.131054-32-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: 5D6E314001C X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 96sqs4b84urmgdkbwzxfb5dcjnajamd8 X-HE-Tag: 1702310304-486406 X-HE-Meta: U2FsdGVkX1/P9JJmyxVaAwpj/dwcwQUniLNpZ8EgR3mKVhDZmhHAWPrt8SoabQmOMurTZ4G5k0dYU17Gr88cSoHolDAd2e1DyEtnP0XvqwUKZV0K7yjei4rW95Fzm7qj4VcbIV1/2Zi93E4NH0H0Ec6fw5RlN74uiuodbYoVqoVxvMsq5l+FghMA9ov+6eCorRGUgdUx8OA08exoF93FFKMNgIeWpo3Bsp6l5fN0UrkUZvfpZcEmW7If1X+5eC+fVhGk3DRk9Aaipn4T0lby1T5r0Vd9vq9cGh5zzz4yfO2Q/jk1nFE/gRZFjPbHPNpzycN7Ob68FNyvMliFKJPfWI5wCuVzGCVXxOJgFJgmLQS0NNQwJfXC7SvQAe1zFAzXKj16ax7X1EvTDQ6VzYkk9HM+Ss3js5NL3S7KJHSJwj5pddGYjO8aeNwUE3iq718KsSrn77Dbl3nb2l88Si5MP7+xX16ftOIoAYP5nFGCTJVEh5j5i/tPRympOkf2h9nPRUkwtwI4BuA8199Yh5t0azf3g719nCT+MsjU+I1Bkykz8BajxAGulnCXFyA5JZQXTHuvbmhniN2BvOaSRqPU8WTQ2fDGG38UCoLgIOoeWEhj+dnz//VrcroYE/gHzJWLjw+TJeeYZJPHmsVbQbdGl4cvRJqAucc7kue1ic4d8+GyTE4495DLI0joHzLglqplz3mw0UkhgNboR1jfCH+6Lt22Gfb2hrF+RCSKaJMVjl3I6E39rHWDa2RcWZmUISaEb+lnNnGKiMvRvkKy5OeZsIXVXV27nxt7hEEOWzfXnQ5RxoFa73jI2qJoEkavkwoTUBPu+K0bMMYX/QRNcmRpufkNXC80GimP0TbVifoC8nmO2LqL1m6MYGpZZ4qLc1xm8Je5w2yRT+qt4ZJfkz8Gk0USsQd9VHe1XWhEDmse35qBQ9kv+R9ThWW/gKFz+Cj6k9os19JpJjSN73LaBlG 1bRvuMmK zufNfkVw2L2Vc7Wv4zKqVwdw9lLbs6aqzkID1ogp0HJbGqp2lzYd63sfYn6cIvattp8HHh9omRMoGGNuQQ4DW447p7fH9KBkWpLEgUhwGFzsJWIdPiKSrzDZI7PYKZtf0mRSlbS5XWbkc1kMB0fFKaFDryb8RNBelwW2PboyeGA0SCEaG56z63GRChdPm8vldJGuk59QUzYNRIF0EHfj+H8c2eQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Refer to folio_remove_rmap_*() instaed. Signed-off-by: David Hildenbrand --- Documentation/mm/transhuge.rst | 2 +- Documentation/mm/unevictable-lru.rst | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/Documentation/mm/transhuge.rst b/Documentation/mm/transhuge.rst index 9a607059ea11..cf81272a6b8b 100644 --- a/Documentation/mm/transhuge.rst +++ b/Documentation/mm/transhuge.rst @@ -156,7 +156,7 @@ Partial unmap and deferred_split_folio() Unmapping part of THP (with munmap() or other way) is not going to free memory immediately. Instead, we detect that a subpage of THP is not in use -in page_remove_rmap() and queue the THP for splitting if memory pressure +in folio_remove_rmap_*() and queue the THP for splitting if memory pressure comes. Splitting will free up unused subpages. Splitting the page right away is not an option due to locking context in diff --git a/Documentation/mm/unevictable-lru.rst b/Documentation/mm/unevictable-lru.rst index 67f1338440a5..b6a07a26b10d 100644 --- a/Documentation/mm/unevictable-lru.rst +++ b/Documentation/mm/unevictable-lru.rst @@ -486,7 +486,7 @@ munlock the pages if we're removing the last VM_LOCKED VMA that maps the pages. Before the unevictable/mlock changes, mlocking did not mark the pages in any way, so unmapping them required no processing. -For each PTE (or PMD) being unmapped from a VMA, page_remove_rmap() calls +For each PTE (or PMD) being unmapped from a VMA, folio_remove_rmap_*() calls munlock_vma_folio(), which calls munlock_folio() when the VMA is VM_LOCKED (unless it was a PTE mapping of a part of a transparent huge page). @@ -511,7 +511,7 @@ userspace; truncation even unmaps and deletes any private anonymous pages which had been Copied-On-Write from the file pages now being truncated. Mlocked pages can be munlocked and deleted in this way: like with munmap(), -for each PTE (or PMD) being unmapped from a VMA, page_remove_rmap() calls +for each PTE (or PMD) being unmapped from a VMA, folio_remove_rmap_*() calls munlock_vma_folio(), which calls munlock_folio() when the VMA is VM_LOCKED (unless it was a PTE mapping of a part of a transparent huge page). From patchwork Mon Dec 11 15:56:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487470 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8A73C4167B for ; Mon, 11 Dec 2023 15:58:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EE17B6B0132; Mon, 11 Dec 2023 10:58:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E94226B0134; Mon, 11 Dec 2023 10:58:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D0D6A6B0135; Mon, 11 Dec 2023 10:58:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id BCB486B0132 for ; Mon, 11 Dec 2023 10:58:25 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 7568DA1AD3 for ; Mon, 11 Dec 2023 15:58:25 +0000 (UTC) X-FDA: 81554994570.14.80892F0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id B061420011 for ; Mon, 11 Dec 2023 15:58:23 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XculKgAT; spf=pass (imf13.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310303; a=rsa-sha256; cv=none; b=gtyWUoMv/gRgah3Jkbt0FCBUbOz7dbzPUVMlFPJCcp90TGuHxjcTGUdmk9iqzd7EkkRZwk aI/O+5LFimr46q6Tlb+rI+DjNtzMNSHcgvbpzbbf88p8qTC3dTb7mMu78T3sEecN6mua34 PGY3KSCw5yHW4SaNwZ+BaOE9/iuN53g= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XculKgAT; spf=pass (imf13.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310303; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yS0UIUzy1DjqGivo1HlVZbIDQrX5/8+loO66ukrqS7A=; b=RWcbudaN79B91A+NmWSsWzfYEfFNUWRml9vraws2tK0MecO70BjMPHKJZUKlNwGLpbH634 AzhXU7tFAxU5APLD1qHfUmgt9Kz/qG/hdnR/p+8WCM5jqqmhBaEmdBj4f6MIhKKd9a29U8 ZVHJqzFdOiDl1U6aJC3MizYJEu/C/R0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310303; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yS0UIUzy1DjqGivo1HlVZbIDQrX5/8+loO66ukrqS7A=; b=XculKgATAgekAXjuuewO75JZcDpT4k4UvArhjSqo+ihJ20RdXhTx2g7ynzh5bhnliIiGlw k4EVHc+/s+eM2meE5vvukIZiITmOq2LxxrFXqwgR9Y85PXcECRrqXQyWA8F+ez381oq27k A5UINgbtCPUwPfh+WVJCgSmPsoSaqyg= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-346-rMmu2rhaOL6tNMECz-xHBQ-1; Mon, 11 Dec 2023 10:58:18 -0500 X-MC-Unique: rMmu2rhaOL6tNMECz-xHBQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 241BA848A83; Mon, 11 Dec 2023 15:58:18 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6AA331121306; Mon, 11 Dec 2023 15:58:16 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 32/39] mm/rmap: remove page_remove_rmap() Date: Mon, 11 Dec 2023 16:56:45 +0100 Message-ID: <20231211155652.131054-33-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: B061420011 X-Stat-Signature: o65s3h7gxqe493y57mm8jjbwgiwantcb X-Rspam-User: X-HE-Tag: 1702310303-118552 X-HE-Meta: U2FsdGVkX1+yud7gvJU5cLZbY0jNkG5Ycp9NVrmyC8vskr6TS6xlxib1kAO4MQGqYrCojwyjVSc9acNcMt6Atxx2p0vpkYfye1rR1phjRsTCt0Jkz6wF2onL2skuWFlMRwDLKmj2KKpnurt3hjIQ89eaNEw55O5nKLXueLb64If6nraJtAD2T2vgecsUqGGvcPFS8YTB/K1a7+5AJtnRa3/Xj9kFHDuxrx6YVdTd4BiaJPbTSRQBNE4u8RJ3052OVqBHr7q2uvPh0lDJf1V2YWmNgxw4USTFRYwx2KL5I/w1/gn8OS8GIKDl9StQjvZLXjJUptvXI781miFe978u37NZA9kNMr2XV1e0D7yZd5rbq9CaWW/t1BrfJYW43bdhLEnKCQvPsRcEXyckmf/TfpeEqKIkteMEUZ8K6+APRQSbqzicFDdxeC2ZLFV47HXYlroJDlxkafC9SRuAD4Z8Y5FknBp1Saj3vFsUDbzLlGp2rJLf75DR8zLdf1WT28YneVUuUO2yePNjgc5+H+UAoXyvf72/va20nJjp9FVdoBKq5G2oI6DJH4P4jRutyXELFuqH6ShRpipkhFjb/FITXc7pcq71AiL3U0uMggK0x5FdrFzVy06idXGhDkRdm1A/F5bUBnWaoOuaw0BVBcn5Sq1Jq6Qjo42ys/5ex/Q/NCMH2Bf59whTzTobEZIyG/3vgnaFUu1i4t3YOHw6g7T+X8l7hJ10l/mhCyPCyEVGf1QbFbC0fiPJSbYvyg2lu+JmM30WdKG0+URLPYh5z9g50mDSb7pDGNoFly/dfn8I3sONQlnP0PDe90Cw5CAMtypp5Ox0CXMojhztsNqlyC0Zs1VVtnqDkIMbCQOP9i2OdMa4Lu1K4wf4iwko/mJ1JtBKsu0LHwDQXfJ+k70nV5Qy6n5L86a3hxLybFJMAZHdDiwDNZOkxUHd1GX1xycCKpvLd1Usry5c8MQhk1IqPxF T7CaLkak U6+Cz0xmk9DOAPrdfAmBlAzbJiQ4fPQIhK71bq+J43bVHEwbr1Q/71ZUm8mdhjhcfdQ2fyxn+aU1Ha/GZCvJRscZLBnTwFeWSdu72PO+bK80yL9SwQGuuBgEIGyrTElld5POKx0aPVzSi98zvX+PDfPN+iSrn6LVU6G7lLNTymbC83NPEFNniWgJeHiriQ56La6cX1wi9hkaqItg1tlJoWlpIj5PO1UrSUQkK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All callers are gone, let's remove it and some leftover traces. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 4 +--- mm/filemap.c | 10 +++++----- mm/internal.h | 2 +- mm/memory-failure.c | 4 ++-- mm/rmap.c | 23 ++--------------------- 5 files changed, 11 insertions(+), 32 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index a266dc0ef99e..0f4eecd03bdc 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -244,8 +244,6 @@ void folio_add_file_rmap_ptes(struct folio *, struct page *, int nr_pages, folio_add_file_rmap_ptes(folio, page, 1, vma) void folio_add_file_rmap_pmd(struct folio *, struct page *, struct vm_area_struct *); -void page_remove_rmap(struct page *, struct vm_area_struct *, - bool compound); void folio_remove_rmap_ptes(struct folio *, struct page *, int nr_pages, struct vm_area_struct *); #define folio_remove_rmap_pte(folio, page, vma) \ @@ -392,7 +390,7 @@ static inline int page_try_dup_anon_rmap(struct page *page, bool compound, * * This is similar to page_try_dup_anon_rmap(), however, not used during fork() * to duplicate a mapping, but instead to prepare for KSM or temporarily - * unmapping a page (swap, migration) via page_remove_rmap(). + * unmapping a page (swap, migration) via folio_remove_rmap_*(). * * Marking the page shared can only fail if the page may be pinned; device * private pages cannot get pinned and consequently this function cannot fail. diff --git a/mm/filemap.c b/mm/filemap.c index c0d7e1d7eea2..beff3865465a 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -113,11 +113,11 @@ * ->i_pages lock (try_to_unmap_one) * ->lruvec->lru_lock (follow_page->mark_page_accessed) * ->lruvec->lru_lock (check_pte_range->isolate_lru_page) - * ->private_lock (page_remove_rmap->set_page_dirty) - * ->i_pages lock (page_remove_rmap->set_page_dirty) - * bdi.wb->list_lock (page_remove_rmap->set_page_dirty) - * ->inode->i_lock (page_remove_rmap->set_page_dirty) - * ->memcg->move_lock (page_remove_rmap->folio_memcg_lock) + * ->private_lock (folio_remove_rmap_pte->set_page_dirty) + * ->i_pages lock (folio_remove_rmap_pte->set_page_dirty) + * bdi.wb->list_lock (folio_remove_rmap_pte->set_page_dirty) + * ->inode->i_lock (folio_remove_rmap_pte->set_page_dirty) + * ->memcg->move_lock (folio_remove_rmap_pte->folio_memcg_lock) * bdi.wb->list_lock (zap_pte_range->set_page_dirty) * ->inode->i_lock (zap_pte_range->set_page_dirty) * ->private_lock (zap_pte_range->block_dirty_folio) diff --git a/mm/internal.h b/mm/internal.h index 222e63b2dea4..a94355e70bd7 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -651,7 +651,7 @@ folio_within_vma(struct folio *folio, struct vm_area_struct *vma) * under page table lock for the pte/pmd being added or removed. * * mlock is usually called at the end of page_add_*_rmap(), munlock at - * the end of page_remove_rmap(); but new anon folios are managed by + * the end of folio_remove_rmap_*(); but new anon folios are managed by * folio_add_lru_vma() calling mlock_new_folio(). */ void mlock_folio(struct folio *folio); diff --git a/mm/memory-failure.c b/mm/memory-failure.c index d8c853b35dbb..01af9295c47c 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -2316,8 +2316,8 @@ int memory_failure(unsigned long pfn, int flags) * We use page flags to determine what action should be taken, but * the flags can be modified by the error containment action. One * example is an mlocked page, where PG_mlocked is cleared by - * page_remove_rmap() in try_to_unmap_one(). So to determine page status - * correctly, we save a copy of the page flags at this time. + * folio_remove_rmap_*() in try_to_unmap_one(). So to determine page + * status correctly, we save a copy of the page flags at this time. */ page_flags = p->flags; diff --git a/mm/rmap.c b/mm/rmap.c index 233432f08e36..b08dd7d6779d 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -470,7 +470,7 @@ void __init anon_vma_init(void) /* * Getting a lock on a stable anon_vma from a page off the LRU is tricky! * - * Since there is no serialization what so ever against page_remove_rmap() + * Since there is no serialization what so ever against folio_remove_rmap_*() * the best this function can do is return a refcount increased anon_vma * that might have been relevant to this page. * @@ -487,7 +487,7 @@ void __init anon_vma_init(void) * [ something equivalent to page_mapped_in_vma() ]. * * Since anon_vma's slab is SLAB_TYPESAFE_BY_RCU and we know from - * page_remove_rmap() that the anon_vma pointer from page->mapping is valid + * folio_remove_rmap_*() that the anon_vma pointer from page->mapping is valid * if there is a mapcount, we can dereference the anon_vma after observing * those. * @@ -1499,25 +1499,6 @@ void folio_add_file_rmap_pmd(struct folio *folio, struct page *page, #endif } -/** - * page_remove_rmap - take down pte mapping from a page - * @page: page to remove mapping from - * @vma: the vm area from which the mapping is removed - * @compound: uncharge the page as compound or small page - * - * The caller needs to hold the pte lock. - */ -void page_remove_rmap(struct page *page, struct vm_area_struct *vma, - bool compound) -{ - struct folio *folio = page_folio(page); - - if (likely(!compound)) - folio_remove_rmap_pte(folio, page, vma); - else - folio_remove_rmap_pmd(folio, page, vma); -} - static __always_inline void __folio_remove_rmap(struct folio *folio, struct page *page, int nr_pages, struct vm_area_struct *vma, enum rmap_mode mode) From patchwork Mon Dec 11 15:56:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487473 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D7C45C4167B for ; Mon, 11 Dec 2023 15:58:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7DFEF6B0138; Mon, 11 Dec 2023 10:58:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 78AA76B013A; Mon, 11 Dec 2023 10:58:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 62C376B013B; Mon, 11 Dec 2023 10:58:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4E80D6B0138 for ; Mon, 11 Dec 2023 10:58:30 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 19165806FD for ; Mon, 11 Dec 2023 15:58:30 +0000 (UTC) X-FDA: 81554994780.22.D1DBA50 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 5370FC0014 for ; Mon, 11 Dec 2023 15:58:28 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=f+GzJMoz; spf=pass (imf22.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310308; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nX1pXuuGCFSgne2iWx/veTQmI+PIa0/pNoky6wKmMd0=; b=kjgaIfh69HgUCAJzlLPuP8SIrCeoTeyvWWpYu00gBNBLRsREKvZnfQbbCq5Ea1pOVnvAqo ZaBNDFkpiU2RTQAl4PiTz8MmH62lQj3zNjd+TsvS2+iDBgJ8+bu00i5QP2OxDowfSb/nvC REfZ4mFBEf9wPXcu0ZJVW6CEs6+7f8Q= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=f+GzJMoz; spf=pass (imf22.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310308; a=rsa-sha256; cv=none; b=JUp/jr9Hba2ShdpofHHFF9kP5lSZ38K9/8cAAoi1etnOKbCWNb0Gd399YeQDh1v4osmhZo G+sKqtbd3adQvVuvlguK6khyNev8S7y/0JRtGVed9gNBT01QsrVZvzucISQnS1WUYtGkIP K3ERMhGtO7HQGAWXLhJUwmsc0l+W+UM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310307; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nX1pXuuGCFSgne2iWx/veTQmI+PIa0/pNoky6wKmMd0=; b=f+GzJMoze76emyCm2Mb/GDC6gnAbAKM5wQfdqmmqRCRj10IBYtW9qrnZzYDTRVuVz7jwC9 5F13jcT1JTaSN9W8tEnz1fbznjuOTk7dYGfDj40RWYyX4sfZAGEMi3wtr82SgcuaCvFYul FwgZu4f6xIu1JOb9E+UG/DFa9YgAeB0= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-515-6fAsG4HZNcWrnusNEcf0NQ-1; Mon, 11 Dec 2023 10:58:21 -0500 X-MC-Unique: 6fAsG4HZNcWrnusNEcf0NQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 5105985A589; Mon, 11 Dec 2023 15:58:20 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5C4221121306; Mon, 11 Dec 2023 15:58:18 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 33/39] mm/rmap: convert page_dup_file_rmap() to folio_dup_file_rmap_[pte|ptes|pmd]() Date: Mon, 11 Dec 2023 16:56:46 +0100 Message-ID: <20231211155652.131054-34-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: 5370FC0014 X-Rspam-User: X-Stat-Signature: wzazcugyhago66d9w3ihnroxd1b37btu X-Rspamd-Server: rspam01 X-HE-Tag: 1702310308-203631 X-HE-Meta: U2FsdGVkX18ff8eTmbg+SK5e9DVbKN2zTehENObiH3S82rFHinoq1LdDsYZcVCd4IBLS1ttLOJOqo/Pe/Sj6oYmGjNILFuf4OmEQzUVu85o5TCgWm+aVPPieRdKN40w61Fc9Q+KVn/HRzZ1mdi93KpTLFbTJgrm/x6GUA5xxPBciT5O7HTAr03tCqFSsA2EDuSgfVowgOxFkaupCl1/BZ2fuhWpOgPCP/ZilthZrxK8bKcDr/HAhGYBdzTTKb1rH6slasJDF3n4VvxuWUzEwkRAow6oMA8xp9IqJu21fPY0Mon3uJL1e5/3bgq735kB2ydK1dRJAU2R5sItR0ADFaoNEAVkW+xNizFrenfLJYKgA8MzVYstxT2gJb/1WS0ovSIsJWCVJ25EW61GSoviGSE544nD+QqFh0DK7Mj0kvnNtNL9EEmfM5ECxs5FFTPNXRNRkubza8RbTUkLbIqNMo2aC6AlVcgAvSdlHCFgS6rkNIQr6yr0zpdnvwolWX9wrDeEMR4lbsTYhyjfDautPLx0LAUJnGa88rO7yJHb2tZLbXQi1gjZcJlYu2q3b9YLPuA41yOFfaY2arUJ23vN2w5X22nVjGxaOCLtAix7H55NkL561ii/0RjDPpIX9G63B+lJJFS0gCj4UwTU3JDvUQJzDLWOLwiOJWn8q0rbd+7KJ+EZqm7JvzaeE/kY1dFf8v6NhfN/nwV1OIC1uV4QCgpVyYDq3oPZUH2yZcaTD86zW6kTpJxb9ajISMINkHaeKWGPMnS+uNpwLTSghOzZPuWlcVAeE7SM2j+GXVQgJrhNG8DUFYDAMjNVR+h4WMuHzd1H2gXXpy45lzarBUF0G1kFKkms9pUuzF1fGElmAgjmAe/d1GrciRh2Mj1xbAW5ePDwP2hkxxyxGrpiHDQk5hF0qHqJdUquztXtHan+6GDWblBrhzehyyXnecwaSIiLzNZiNiF8TPw/qyExIUaJ sMgwkSn1 cNtw1UahNLVSmKvw8AN/qJCtbV9tV6VOoLwQbw9eIAunfgn8cH9IP7PtzEN1R8pdv+/JbS2TWBe1y9HuX0gfLtZWahTEr6K7GNOJiAgeLFGmsDLGslQiCd6EL2FXcgvFx3/3M43nYVk0aLegkF+sS3u79d+/TzCjWGV4DEK/285R1D60hZ15w+Rk9TKpx52tq2GZcqZ//rUA5yblPzfYuP3YwgxOpedeuzr/5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert page_dup_file_rmap() like the other rmap functions. As there is only a single caller, convert that single caller right away and remove page_dup_file_rmap(). Add folio_dup_file_rmap_ptes() right away, we want to perform rmap baching during fork() soon. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 59 ++++++++++++++++++++++++++++++++++++++++---- mm/memory.c | 2 +- 2 files changed, 55 insertions(+), 6 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 0f4eecd03bdc..df60e44fecad 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -311,6 +311,60 @@ static inline void hugetlb_remove_rmap(struct folio *folio) atomic_dec(&folio->_entire_mapcount); } +static __always_inline void __folio_dup_file_rmap(struct folio *folio, + struct page *page, int nr_pages, enum rmap_mode mode) +{ + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); + + switch (mode) { + case RMAP_MODE_PTE: + do { + atomic_inc(&page->_mapcount); + } while (page++, --nr_pages > 0); + break; + case RMAP_MODE_PMD: + atomic_inc(&folio->_entire_mapcount); + break; + } +} + +/** + * folio_dup_file_rmap_ptes - duplicate PTE mappings of a page range of a folio + * @folio: The folio to duplicate the mappings of + * @page: The first page to duplicate the mappings of + * @nr_pages: The number of pages of which the mapping will be duplicated + * + * The page range of the folio is defined by [page, page + nr_pages) + * + * The caller needs to hold the page table lock. + */ +static inline void folio_dup_file_rmap_ptes(struct folio *folio, + struct page *page, int nr_pages) +{ + __folio_dup_file_rmap(folio, page, nr_pages, RMAP_MODE_PTE); +} +#define folio_dup_file_rmap_pte(folio, page) \ + folio_dup_file_rmap_ptes(folio, page, 1) + +/** + * folio_dup_file_rmap_pmd - duplicate a PMD mapping of a page range of a folio + * @folio: The folio to duplicate the mapping of + * @page: The first page to duplicate the mapping of + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock. + */ +static inline void folio_dup_file_rmap_pmd(struct folio *folio, + struct page *page) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + __folio_dup_file_rmap(folio, page, HPAGE_PMD_NR, RMAP_MODE_PTE); +#else + WARN_ON_ONCE(true); +#endif +} + static inline void __page_dup_rmap(struct page *page, bool compound) { VM_WARN_ON(folio_test_hugetlb(page_folio(page))); @@ -325,11 +379,6 @@ static inline void __page_dup_rmap(struct page *page, bool compound) } } -static inline void page_dup_file_rmap(struct page *page, bool compound) -{ - __page_dup_rmap(page, compound); -} - /** * page_try_dup_anon_rmap - try duplicating a mapping of an already mapped * anonymous page diff --git a/mm/memory.c b/mm/memory.c index 9a5724cf895f..42a0b7b41b86 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -965,7 +965,7 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, rss[MM_ANONPAGES]++; } else if (page) { folio_get(folio); - page_dup_file_rmap(page, false); + folio_dup_file_rmap_pte(folio, page); rss[mm_counter_file(page)]++; } From patchwork Mon Dec 11 15:56:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487472 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD6D6C4167B for ; Mon, 11 Dec 2023 15:58:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CBE996B0136; Mon, 11 Dec 2023 10:58:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C6C336B0138; Mon, 11 Dec 2023 10:58:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE7A26B0139; Mon, 11 Dec 2023 10:58:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 9B6886B0136 for ; Mon, 11 Dec 2023 10:58:28 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 6812F40784 for ; Mon, 11 Dec 2023 15:58:28 +0000 (UTC) X-FDA: 81554994696.10.EBB938B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 4201714001B for ; Mon, 11 Dec 2023 15:58:26 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CZj3GLqm; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310306; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hi7b3teT5yay/0ynlIbF8dSmjMX9HoAHjoRGBB4rTLU=; b=Lfqap0AACvdBIUPf8S43DGf7TdX6J7o0d/5cWW0euRXwnPnva73yqSg5pdaMCxf65HHp5a WsneqOlRTo3rcwoimVqDQdTOl1NdtK2D94TGxX1Dx3O1Zq6gleNLnt7Z7bn3fqctPlT8Ox 2Yyj5Czx1IlPz5mdnpfsKg1h1eKHa/k= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CZj3GLqm; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310306; a=rsa-sha256; cv=none; b=bto3tBd12zmPbPJz5Tc2xXIW7TSnl9iwyQJSL6KiYL4wEU+ACHFUQH3lPfKS7Di5f+SPYH niElIXPKNnyuD+MR4vq7iE26vJwFo94DclAR8J/bFungMrFaLZsNwSVYK7msastEJwi0JO jIYyVW4a5Jws7zTWXEqkCeXPXOZVz08= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310305; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=hi7b3teT5yay/0ynlIbF8dSmjMX9HoAHjoRGBB4rTLU=; b=CZj3GLqmedQG+dD0ScWar45u6K+kcaijkbq8LCv8kF1qDL9Y2J5/vZm7ZC0S4Pz59ErPM3 33YiVtj5lHT38sHnwQD9m68mZPiIV9UJhm2zpkx2i4+vDeawIet0A5IHOoQFgHaBtY3Y77 vfoxK48a2pdjCT3iaSO1yQSftTzqB+g= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-645-vRbs79ACOCeF2HWItzmhcA-1; Mon, 11 Dec 2023 10:58:23 -0500 X-MC-Unique: vRbs79ACOCeF2HWItzmhcA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C070D86301D; Mon, 11 Dec 2023 15:58:22 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id B14EA1121306; Mon, 11 Dec 2023 15:58:20 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 34/39] mm/rmap: introduce folio_try_dup_anon_rmap_[pte|ptes|pmd]() Date: Mon, 11 Dec 2023 16:56:47 +0100 Message-ID: <20231211155652.131054-35-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: 4201714001B X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: te55p5mt6rt4nra9cr4jmmczq7ripu5m X-HE-Tag: 1702310306-657019 X-HE-Meta: U2FsdGVkX18ZpgeBficVY8HMH9hiwkUj7r102vw++yf/XaKvmlO7onP8f5Uwq5dwrvxP/D1k6P/EfjwbtlyNuFa1/Bs7SH7QaF1riPNsHe1/qa+6agZCnipX5wb2PWHW+LRxZk8WuYCgzQxvC2IvsVbwoVW+QTWP2h/9UuDMYvbmFQYBdb/WhGeXOOb3Z3qCJTfdNBg7AhXxesTtP8VNhMxLkNxIxqCD/8J7x6LHnqhIgeSKxY05Jrr8sqVxR90UNQ82zp6h1zIb8TTUdw9cshCEUJAL5laRl/ZbvU5L5d//brzCXkcjYJRaaWyKVObRueQjrnkOM/LOgiuv3WmqgyWxC3fUZf4/w89lG+uA2Qg7J2O4Bj0m7VvsdraTsKNvD6T3F+HVHlM/qX6MOOGpKGXbXO3QD56Y4LxAVV2jj9CoEqdiRvMkm0XvqQoDFESMbNO2lbm5/BvlDCzOTGnVNU3+iZaj1XfRCImy2afKH0ylnr6hUqx1KttouQTZ7gYCScyQ6/pPiicfhysakTj4muEeO1N7fA7GZjq3NyGis2B0Mlmli93COg6wbjRP53hIazg6hOnQ48CwwhoTsvZBjjiJwT8ceezE1cS8nJQXdHcF0enKCJ9IIExPzvPLrrqc6R/tGmur6zwJZOWdo+KSwoRuBsDBJ93o5+etPfVze7/9JT1HhpdBzwO9TL+/2oPVQUfXOxsAZ34Z5MUZlkFojWzCxGyEEcDq+3DhSvRY8I7uu6gM/D5tmUXPDWB5TK4ShR/FQAy6tVvWJZIrJpbLD/kitDyFdGUO+qSa56jfvwJgvHFLt2MjrMftA9eKpPPyqsQlT3yD/O6f49zkmShSgxlXb3KVwiU6/Wq0VeIVDujgBk7TjchC54QJCVykXi9wkarIn6LrUphdw6Awn2ZXHIwEfmpzMLfINmTlgsgfottpQg3ww2FZX4J0oNmKyh7f2aU0RUGAkEm+yXLr3dB kTpdG0ee or5Epp/tWWiXmYzgr/AcygKuVuYQ6bplJ0GT20Xu/NZA1qU28lQT0b/7kWddDL7GrOAGLd9cK7JvTCIpTprgAb9WCTzeXP4qqpwsBEYvwwqbI8GvmAQ1eTgNaZ0lHMrMFmqOqNFfLP1xxNu1mLafZEBNDGrLRZR/7VU+IM2QvRJGuBOW82Mlr+qbL5bQhto/dmigKMFmNc5wFRLH0EvpmGH+FvdGSR50kesFL X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The last user of page_needs_cow_for_dma() and __page_dup_rmap() are gone, remove them. Add folio_try_dup_anon_rmap_ptes() right away, we want to perform rmap baching during fork() soon. Signed-off-by: David Hildenbrand --- include/linux/mm.h | 6 -- include/linux/rmap.h | 150 ++++++++++++++++++++++++++++++------------- 2 files changed, 106 insertions(+), 50 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index ae547b62f325..30edf3f7d1f3 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1975,12 +1975,6 @@ static inline bool folio_needs_cow_for_dma(struct vm_area_struct *vma, return folio_maybe_dma_pinned(folio); } -static inline bool page_needs_cow_for_dma(struct vm_area_struct *vma, - struct page *page) -{ - return folio_needs_cow_for_dma(vma, page_folio(page)); -} - /** * is_zero_page - Query if a page is a zero page * @page: The page to query diff --git a/include/linux/rmap.h b/include/linux/rmap.h index df60e44fecad..c6d8a02ecd56 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -365,68 +365,130 @@ static inline void folio_dup_file_rmap_pmd(struct folio *folio, #endif } -static inline void __page_dup_rmap(struct page *page, bool compound) +static __always_inline int __folio_try_dup_anon_rmap(struct folio *folio, + struct page *page, int nr_pages, struct vm_area_struct *src_vma, + enum rmap_mode mode) { - VM_WARN_ON(folio_test_hugetlb(page_folio(page))); + bool maybe_pinned; + int i; - if (compound) { - struct folio *folio = (struct folio *)page; + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); - VM_BUG_ON_PAGE(compound && !PageHead(page), page); + /* + * If this folio may have been pinned by the parent process, + * don't allow to duplicate the mappings but instead require to e.g., + * copy the subpage immediately for the child so that we'll always + * guarantee the pinned folio won't be randomly replaced in the + * future on write faults. + */ + maybe_pinned = likely(!folio_is_device_private(folio)) && + unlikely(folio_needs_cow_for_dma(src_vma, folio)); + + /* + * No need to check+clear for already shared PTEs/PMDs of the + * folio. But if any page is PageAnonExclusive, we must fallback to + * copying if the folio maybe pinned. + */ + switch (mode) { + case RMAP_MODE_PTE: + if (unlikely(maybe_pinned)) { + for (i = 0; i < nr_pages; i++) + if (PageAnonExclusive(page + i)) + return -EBUSY; + } + do { + if (PageAnonExclusive(page)) + ClearPageAnonExclusive(page); + atomic_inc(&page->_mapcount); + } while (page++, --nr_pages > 0); + break; + case RMAP_MODE_PMD: + if (PageAnonExclusive(page)) { + if (unlikely(maybe_pinned)) + return -EBUSY; + ClearPageAnonExclusive(page); + } atomic_inc(&folio->_entire_mapcount); - } else { - atomic_inc(&page->_mapcount); + break; } + return 0; } /** - * page_try_dup_anon_rmap - try duplicating a mapping of an already mapped - * anonymous page - * @page: the page to duplicate the mapping for - * @compound: the page is mapped as compound or as a small page - * @vma: the source vma + * folio_try_dup_anon_rmap_ptes - try duplicating PTE mappings of a page range + * of a folio + * @folio: The folio to duplicate the mappings of + * @page: The first page to duplicate the mappings of + * @nr_pages: The number of pages of which the mapping will be duplicated + * @src_vma: The vm area from which the mappings are duplicated + * + * The page range of the folio is defined by [page, page + nr_pages) * - * The caller needs to hold the PT lock and the vma->vma_mm->write_protect_seq. + * The caller needs to hold the page table lock and the + * vma->vma_mm->write_protect_seq. * - * Duplicating the mapping can only fail if the page may be pinned; device - * private pages cannot get pinned and consequently this function cannot fail. + * Duplicating the mappings can only fail if the folio may be pinned; device + * private folios cannot get pinned and consequently this function cannot fail + * for them. * - * If duplicating the mapping succeeds, the page has to be mapped R/O into - * the parent and the child. It must *not* get mapped writable after this call. + * If duplicating the mappings succeeded, the duplicated PTEs have to be R/O in + * the parent and the child. They must *not* be writable after this call + * succeeded. + * + * Returns 0 if duplicating the mappings succeeded. Returns -EBUSY otherwise. + */ +static inline int folio_try_dup_anon_rmap_ptes(struct folio *folio, + struct page *page, int nr_pages, struct vm_area_struct *src_vma) +{ + return __folio_try_dup_anon_rmap(folio, page, nr_pages, src_vma, + RMAP_MODE_PTE); +} +#define folio_try_dup_anon_rmap_pte(folio, page, vma) \ + folio_try_dup_anon_rmap_ptes(folio, page, 1, vma) + +/** + * folio_try_dup_anon_rmap_pmd - try duplicating a PMD mapping of a page range + * of a folio + * @folio: The folio to duplicate the mapping of + * @page: The first page to duplicate the mapping of + * @src_vma: The vm area from which the mapping is duplicated + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock and the + * vma->vma_mm->write_protect_seq. + * + * Duplicating the mapping can only fail if the folio may be pinned; device + * private folios cannot get pinned and consequently this function cannot fail + * for them. + * + * If duplicating the mapping succeeds, the duplicated PMD has to be R/O in + * the parent and the child. They must *not* be writable after this call + * succeeded. * * Returns 0 if duplicating the mapping succeeded. Returns -EBUSY otherwise. */ +static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio, + struct page *page, struct vm_area_struct *src_vma) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + return __folio_try_dup_anon_rmap(folio, page, HPAGE_PMD_NR, src_vma, + RMAP_MODE_PMD); +#else + WARN_ON_ONCE(true); + return -EBUSY; +#endif +} + static inline int page_try_dup_anon_rmap(struct page *page, bool compound, struct vm_area_struct *vma) { - VM_BUG_ON_PAGE(!PageAnon(page), page); - - /* - * No need to check+clear for already shared pages, including KSM - * pages. - */ - if (!PageAnonExclusive(page)) - goto dup; + struct folio *folio = page_folio(page); - /* - * If this page may have been pinned by the parent process, - * don't allow to duplicate the mapping but instead require to e.g., - * copy the page immediately for the child so that we'll always - * guarantee the pinned page won't be randomly replaced in the - * future on write faults. - */ - if (likely(!is_device_private_page(page)) && - unlikely(page_needs_cow_for_dma(vma, page))) - return -EBUSY; - - ClearPageAnonExclusive(page); - /* - * It's okay to share the anon page between both processes, mapping - * the page R/O into both processes. - */ -dup: - __page_dup_rmap(page, compound); - return 0; + if (likely(!compound)) + return folio_try_dup_anon_rmap_pte(folio, page, vma); + return folio_try_dup_anon_rmap_pmd(folio, page, vma); } /** From patchwork Mon Dec 11 15:56:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487474 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74C9FC4167B for ; Mon, 11 Dec 2023 15:58:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D35F26B013A; Mon, 11 Dec 2023 10:58:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CBBA26B013C; Mon, 11 Dec 2023 10:58:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC0306B013D; Mon, 11 Dec 2023 10:58:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 9B0EF6B013A for ; Mon, 11 Dec 2023 10:58:32 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 7CA38A0797 for ; Mon, 11 Dec 2023 15:58:32 +0000 (UTC) X-FDA: 81554994864.06.8D05D9D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf18.hostedemail.com (Postfix) with ESMTP id C06791C001C for ; Mon, 11 Dec 2023 15:58:30 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NLo2j4Rx; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310310; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dpwgLE5BjdH1zvm+T58sLO1pe3sX/Qw4UhxJ1rV6Cd4=; b=N7V5JUPUCX9AccXHSQDEGkpWHZXkQDuH2yLrhozLTf92lItxxXXpcgHcTbYnPQEDyVSu3O IAzi6FL5UusG8bZlBhMMNOAYcc6CgUe7x5ktBQNHQs1DSwXtZ2LkQ1Gyq0e4aP7JBCEiMB 8IUQil5yNbctBjtKfehNHO+AClhcATY= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NLo2j4Rx; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf18.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310310; a=rsa-sha256; cv=none; b=ubLy/QctQkkz7EwqfESjQjbbm6ohIRskzDlktE8QXKitojXddSiBaEucDS6k6oPdCDK1nO TKHaSCod53EpUhvUlZhxuX5meZNWJHydw/5CUK+qNfnMHsQF95GKV7iHOsp5VpKRoClXou U8sHjPKIYDBwmJr268hazDyucOuJfUE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310310; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dpwgLE5BjdH1zvm+T58sLO1pe3sX/Qw4UhxJ1rV6Cd4=; b=NLo2j4RxsL9KZUyAJubER44eVS0Bm2Kc1uePKjB8v0WzPzplA7mqNHhEZcGo6SoZbakhFk gHQTBeXzBxcwfRna9WMNb2GFTQM8pjT+vyQy+gZHXr2l9c6S5ny9wpaTN4SVe7o9PPCKHw AWf8bDC50+kLaQ3HPDyiZXByb3QzVis= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-410-c8kAgQI6P8SQgJ12PSpYrA-1; Mon, 11 Dec 2023 10:58:25 -0500 X-MC-Unique: c8kAgQI6P8SQgJ12PSpYrA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 4CCFB870834; Mon, 11 Dec 2023 15:58:25 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2E9131121312; Mon, 11 Dec 2023 15:58:22 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 35/39] mm/huge_memory: page_try_dup_anon_rmap() -> folio_try_dup_anon_rmap_pmd() Date: Mon, 11 Dec 2023 16:56:48 +0100 Message-ID: <20231211155652.131054-36-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: C06791C001C X-Stat-Signature: n4o5k3oc6ap3yhq3eewidx4yqgppu6w5 X-HE-Tag: 1702310310-73069 X-HE-Meta: U2FsdGVkX19YIDwdlRleZ6rk6Li1DmZT8a8cb842Qpca72QSgRXZfiZtkdJ675hS8Z8IuExDaMAwb5yH0BeYILlGIyyk+wWp3HFHBlk6SlkUhC5EvIKuj50SLywSQgqet6ATO5tNtRq6IrlWnDcAuGbajjuuMCXa74zfkokF5UPUebSKnUeNTfALAELdk9eTKjpLDMIscT/gNtXt3s0e/ET5Jrk7u3P2QWzWdy/tohQ/cDQVUStx0fnozw/vrU56jIbPW/4wZsrOq3jZ4fwEdvCkA0NHRkvLFGo6VMGa6QmbCoDjY2VuWU7y1kJj1v1f0P3FtND5eztegeFRRAVDDDDjnHwDPfSYnCeIMepiyJx4EfDp3pjbNktKDNp56dEcAPHAWC95mpC4JA6iEa84HxCHQ5C1MXZW6hsgSEG7u3hXmMeX+TY2vf4ZO9r6txzZgnlrJuviglluEuktEahcAyA05ulEWLjpc8WyOrAhJAPmq7pZKYbELnPJxb1KaP06+srewGBrtaOm/duF/gKmuoPb5GvRNKN2qDYRzYbQknivJ1/YQhxSOBQvwd2OiV2fxaasXHiQO5C8nvAPZerVFk/mxARxUgDsviwuTrzZmDm+1cSEd8dmqxYU9/v4h7mOvP6nPwIx/hPKuvU4PXEwBtKG4YOubN9G3yrXTeOiOwiRZe1VsmlF0yitQqB1xNBm+g1Er8ndypfFtvFsol1WT5MPsshoZ6/HnluO6qMIk+yJtFJcMciIySgpc5ModQlXRI8rsqnkuLbRisISajAwiit15loyycW+g6tjPXUUicEIsUXG72MZXhpYLHWtQdRWjk5X3VipoCg/CzUex3VoYQbBkdXS0FjYbMZmcKVqbGVeodXobnyh7XkWin4PgrgGXfvbx//M+v4Sqgg3wsqb3j7V4asZKqRe4qVFGzuNfDfpQLCaAxgOuR8RNra+MGhmIqlia1HJYEC7Y1tp6HC X4PHs0mR ZQ2czrXWJDnmZTeKAm9R6ws0yYF60oXdoFUOfjeBsUQkVxnntliA+Vxr44Z8rEjJ74BciiRv/2qonwu+0HAAEe/KFBKjUCQp3rvSxNhtlFgYjhyp227bKUq220PrbMUUIBa+weP3c/5Pj/XE4TBfDDEPqDO/kgnjCIG+jwxBdSch1AUh5eD24VfRiOmX9IgNA5xLug0kMDaQCbpTptx1BvaWCyQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert copy_huge_pmd() and fixup the comment in copy_huge_pud(). While at it, perform more folio conversion in copy_huge_pmd(). Signed-off-by: David Hildenbrand --- mm/huge_memory.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index cfaa8b823015..34f878916621 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1275,6 +1275,7 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, { spinlock_t *dst_ptl, *src_ptl; struct page *src_page; + struct folio *src_folio; pmd_t pmd; pgtable_t pgtable = NULL; int ret = -ENOMEM; @@ -1341,11 +1342,12 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, src_page = pmd_page(pmd); VM_BUG_ON_PAGE(!PageHead(src_page), src_page); + src_folio = page_folio(src_page); - get_page(src_page); - if (unlikely(page_try_dup_anon_rmap(src_page, true, src_vma))) { + folio_get(src_folio); + if (unlikely(folio_try_dup_anon_rmap_pmd(src_folio, src_page, src_vma))) { /* Page maybe pinned: split and retry the fault on PTEs. */ - put_page(src_page); + folio_put(src_folio); pte_free(dst_mm, pgtable); spin_unlock(src_ptl); spin_unlock(dst_ptl); @@ -1454,8 +1456,8 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, } /* - * TODO: once we support anonymous pages, use page_try_dup_anon_rmap() - * and split if duplicating fails. + * TODO: once we support anonymous pages, use + * folio_try_dup_anon_rmap_*() and split if duplicating fails. */ pudp_set_wrprotect(src_mm, addr, src_pud); pud = pud_mkold(pud_wrprotect(pud)); From patchwork Mon Dec 11 15:56:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487475 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E626C4167B for ; Mon, 11 Dec 2023 15:58:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AC2D56B013D; Mon, 11 Dec 2023 10:58:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A708B6B013E; Mon, 11 Dec 2023 10:58:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C4B26B013F; Mon, 11 Dec 2023 10:58:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 784C26B013D for ; Mon, 11 Dec 2023 10:58:33 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 4FC3240727 for ; Mon, 11 Dec 2023 15:58:33 +0000 (UTC) X-FDA: 81554994906.28.1ADBA0C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 9053C40022 for ; Mon, 11 Dec 2023 15:58:31 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ikcVbtBe; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310311; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XplrMRug9dV6L5N0sR7KY1Q40igMgZePuFWGdErJCwc=; b=WT6itKPtqtZ9vy1z6X1SJoFHn0BB5KFN9394T/MTQ+UV4wqO6t1ZDNX4zpl2Ft5WnSuI/c sVdgZB5g8+xcGLJ9KhC+MVCibAIn48HgslLGy542dF2fz8Pal+gRPhY3sBn2ZcDIohdoK0 ZASSAXhodScqsKI75Ykqbh5I7KjDKkg= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ikcVbtBe; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310311; a=rsa-sha256; cv=none; b=qx889YrZVF5cuYi/uXn7mPgXyCYPZttra2knXS2pm8fUbxC2Mddp+QTzx1rBcugbpbrjVR GLM/rHDjcz9mOvl0dApSQGpCgPJmuDbFU7/yeKt2QQZjm9VuUKSa6orLE6SRT2fBWdYbfn QAWTOG9mZRtL4zLRm8DVUxF+bCY8MA4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310310; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XplrMRug9dV6L5N0sR7KY1Q40igMgZePuFWGdErJCwc=; b=ikcVbtBeT2o9jvmwKB4OOtngV2lM77eI0VRCJwtH2GA1PUSA+KXsP4FFttNtDg1yCSP32D NFb1MPnGWWmTF0SCLP2or81upBv890UMTh4f57wtCZ5xwReDJTS0+Hg0ZYW7Urb36vqUof unNrsljuYqSehmj60EJEf8Pj9sGnc4Y= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-94-z8fIYRY_PHalWsMPzvXDSw-1; Mon, 11 Dec 2023 10:58:28 -0500 X-MC-Unique: z8fIYRY_PHalWsMPzvXDSw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3551C185A793; Mon, 11 Dec 2023 15:58:28 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9BFAB1121312; Mon, 11 Dec 2023 15:58:25 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 36/39] mm/memory: page_try_dup_anon_rmap() -> folio_try_dup_anon_rmap_pte() Date: Mon, 11 Dec 2023 16:56:49 +0100 Message-ID: <20231211155652.131054-37-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: 9053C40022 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: cmnnqbg9x63xq49bpz3mz4mspaz1p75h X-HE-Tag: 1702310311-644520 X-HE-Meta: U2FsdGVkX1/BvoiRfLx1hndEKOQivDEEmcpXwbfKAl+inqWQVYkptLLkRtRvqXQga2HnHljDSnQ9WhykaKzILyraHD7GXU/fP2hOXwYtz+UlVQY+CNE27N/GTeub3qLC8BevdDlEBSTb0N9OIGuDu3SdQHf1GC0wCa+2qNeTL47+lN9kK+psm6FvfmcdVTJlelMdu66i17EWdXCN3TpdkONdSq6vrpjcFVa/IPfLyxCn+/JG6JXj+ceQLltpvNnlruvzB+BeKl3BCfBgAxquP1odmV+nvQXdmq/VEh+uRlUkOv5gz9ZECHO2tpzIEXdI6A4ds0ffyJbBJqqB1yTe9fgbvLyLkP60W6fL6SWSI2qjC0MTrC4ZLjo7IUeI+XHvsSz6XuZhEzQ56r75d51OYHiX+BTJI7yOmFzRjH7fqEFIQz5/R8pIhS6q+d5KAHbUF/hzgGez9HE3lv9Xx0jw7MeMhAttXgvcIoL3bJxNXhwdm4mQ51f+LZsf8n2SADb3xFLbCeAoaND4zD7JZYmNxDmz4eR5Fz9nLLKwqk5OT2oRuPo/Xeuw7msL8j365z3TJ9L1Hx4QeZ3CIO2yTtHJGeQ4gCrYsbcap9eWkzyy9KEPh8PlfD1gJmHcytC7EOSVM8Qa0X3iNsNJECdEyzXEFHPId3cArFpp6UONnsu/0JKvE2XNBUbeOhAFfJq4fKQ4i3v5qAA8qKa0QTBUvet+ZZ7e6ByMkNr3OI1lX6cXsF2V2Zpx6cELRD924k6lxfCXQMG8dMFAYMBJXwxBbCZu6zaR/Fnxe+Cd72dtqhHZhDYig00Xnf/2yapF8hx4yderqv8a2feizkvZZiOIIfVuP4HX9cMFjPqMXWOhxNm44VPXfO3AZZZYgHL3tR2akJfwJ7cXPTBmUNyHi9ylrhpJJBcdZpJTW0lh0BT/85qnfQD53D5DJMzh3OVayOC2SPJBhgC2IxfCp8yeM+RElZV 6mXNnhqX 5q34Tf8mrdUapvmd0prGyRC1BW3pl/DjCI1jCJjHoysFgsB/Fs98KUnQccb86dVtt1ukYKYsjgrZ0plGknK/c1QnxVQhTQ7/PY69eKUe6zWwCtmSqcdFlVl9nYqaTQ7rPrvXY1y7wSd3hg1KItr+iDucZ10LTPkT8Oe3cNnMYIDCSpiRCL4ZpZ67rp6NI0WGi61RibG+bwQDPEMyUCt+r7hz0/w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert copy_nonpresent_pte(). While at it, perform some more folio conversion. Signed-off-by: David Hildenbrand --- mm/memory.c | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 42a0b7b41b86..caaf4add6fa2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -785,6 +785,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, unsigned long vm_flags = dst_vma->vm_flags; pte_t orig_pte = ptep_get(src_pte); pte_t pte = orig_pte; + struct folio *folio; struct page *page; swp_entry_t entry = pte_to_swp_entry(orig_pte); @@ -829,6 +830,7 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, } } else if (is_device_private_entry(entry)) { page = pfn_swap_entry_to_page(entry); + folio = page_folio(page); /* * Update rss count even for unaddressable pages, as @@ -839,10 +841,10 @@ copy_nonpresent_pte(struct mm_struct *dst_mm, struct mm_struct *src_mm, * for unaddressable pages, at some point. But for now * keep things as they are. */ - get_page(page); + folio_get(folio); rss[mm_counter(page)]++; /* Cannot fail as these pages cannot get pinned. */ - BUG_ON(page_try_dup_anon_rmap(page, false, src_vma)); + folio_try_dup_anon_rmap_pte(folio, page, src_vma); /* * We do not preserve soft-dirty information, because so @@ -956,7 +958,7 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, * future. */ folio_get(folio); - if (unlikely(page_try_dup_anon_rmap(page, false, src_vma))) { + if (unlikely(folio_try_dup_anon_rmap_pte(folio, page, src_vma))) { /* Page may be pinned, we have to copy. */ folio_put(folio); return copy_present_page(dst_vma, src_vma, dst_pte, src_pte, From patchwork Mon Dec 11 15:56:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487478 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCDE0C4167B for ; Mon, 11 Dec 2023 15:58:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5B86E6B0142; Mon, 11 Dec 2023 10:58:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 547116B0144; Mon, 11 Dec 2023 10:58:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 397F46B0145; Mon, 11 Dec 2023 10:58:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 22D486B0142 for ; Mon, 11 Dec 2023 10:58:40 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id E3963806F2 for ; Mon, 11 Dec 2023 15:58:39 +0000 (UTC) X-FDA: 81554995158.24.F83E1A1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf11.hostedemail.com (Postfix) with ESMTP id 1EAF940018 for ; Mon, 11 Dec 2023 15:58:37 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DTzUNQUA; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310318; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Bl7kAOkMik26idrH9JyJnR9t8BI/kVxIDTOYx+vyhms=; b=H1ZyqFMVD2YAnSe6CV+dx7wgNxB0YiJyzRTLZVwgRzs04JsmJUW1kxRuwFGUP35g51FPM4 8mzbetb8iXnqZKHM2wIRenfJGWIlXlN6Gm3XHHS0hj6xVCzXsvg+P09dtI63ZYejzLWyhs Jv1wxYE2pwaq0PtsQoqYuYOOtaX82do= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DTzUNQUA; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310318; a=rsa-sha256; cv=none; b=CyV3/nPh/LbfnXhrNRPD4/9j8qIVDgy49/cCpCLHc4douMcrYlHN5mwWyvQbFQhLU2nQLn 2fC8I8abN0gGe+y2eJM39zH8hrTGxLfw+uQqMVljWU0nM1BqmSvdOxhdwzl5aOceQsE4QO FtSb8r3aCjVTtCtVWXWjJIDUnU8upZY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310317; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Bl7kAOkMik26idrH9JyJnR9t8BI/kVxIDTOYx+vyhms=; b=DTzUNQUAUT6rGI35uEqjRNLq5avGqAPz9XzN+bxFGv3gRtbNtil4nXsEPjcfXABMHidKJQ NF7o7lrMFHmoch3GF4XKcJpmOWi4IYdp6wsud6Y4Is8yWrxCfNEFvwggkmB+h3EbU9fRFM A4BeN/xZJZHgubdkx7k5yc9UvYOvdMc= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-382-bVYLacyzO7-p7IF_oXo34w-1; Mon, 11 Dec 2023 10:58:31 -0500 X-MC-Unique: bVYLacyzO7-p7IF_oXo34w-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D01CC280640D; Mon, 11 Dec 2023 15:58:30 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 945CB1121306; Mon, 11 Dec 2023 15:58:28 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 37/39] mm/rmap: remove page_try_dup_anon_rmap() Date: Mon, 11 Dec 2023 16:56:50 +0100 Message-ID: <20231211155652.131054-38-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 1EAF940018 X-Stat-Signature: 5h9wfra7xumz35z3ccdx9o5ddgx7wdms X-Rspam-User: X-HE-Tag: 1702310317-972181 X-HE-Meta: U2FsdGVkX18uIfLBBzsYrosf2qe4e47T4ruczDzQXNGSDLEI/j5PM/BA/MXI/l/fg5dnIz+TdaEmt1KpHSVIh0X5uJgD/frEUjDVr2/hJUnZM+45bgzEFvzlceZgBuqZhwTg5JDMkTEnCvZxfQht5B3UXD3Q9Oxm/uF6kV6I4K1oN7TLE0U2ItgsFafv/Vff1q3q2OFDqDIu1iYK7H03x4PxDLuZeyvEFviXxcbhVVnBeCHZRBL+5lSYno24H/FbjEvSDASnMAy9D6MFViV0jvBSG1KEWT0NkpgoC8Ml5Q822UIn9L8sRjRetPhWvKzm3u93/nHfk+lONdbv/W457xeeaTQk3jHNncsV1pZlzOQlCH6aXjK5HEANKURRD8kkvPpxRzhl2w5Yxbko0WWqnn6g/PmmGC5vt/trAcgI3f0jRAGnNbLe39ovM3sh7GO7K0yZAcs1VDQZviCnzGLE7HaQGJfxvd9xohv+3EatSE0dL225EoWoIoGzYAwYvgaJQqmfxXyinc8JIFh3lqRc5MckkVwfTMJ82NadHXPRThlEFnSHkScPKId4i38Pl3JLSjSJmlh8WiZyIR9+auUp7YeYgr55o/x7dY6ezxtYiymLwqlKsF7kISAofGEg7yTAaP27Ab+aHcCtOnfDTKxlRqkNPsXRAxspBefJXuDRXkxbjsg9N/35DwvQKc5F56cbxojynkjcg1krwfWcqxPVYV3BDN+pAAHvhJr39f98CxKYZolTKshB3B8oNc/zpx1Q0U6rezj4RawzpxpFaZH8HZoPcXATDCmyBxE2moCyBnoI5OOQpsuV09ZD2z6nWxtT2eo7cKQx3gvpwM1LQljl2xm2l9XbGm8F9l2fwujSl1y1i51jr00cVx9Tj6YV+kSCR/YxwlK76y4kyx+TJuryW9rUFq0ZQCGjxbXH30/qKCs/5pXulGvjJ6nzzi3iZQS4TDTG5chsYT40u8tpZOg r4aRMCm3 ofYcvMRk+sz6fG0fvMiSc/vmgnNbuYhwf++klcHsQ/lfpCBPiFlclKPX40Vw+qrSoAkl4wt9ECiH9PaAbxZES0XyFsf5N1sLe0L+1RvdjLuYESx3sqr6pyuoRZlXAC7XWiFrcCT+IOMq3t9cAFKeWhwy6P6lNpnWDVc3zCoyXHH5S7PI4dhAD7t5jFLJirxkEJ3UL7QW42mKD48Ho4iR+ObJ+ww== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: All users are gone, remove page_try_dup_anon_rmap() and any remaining traces. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 16 +++------------- 1 file changed, 3 insertions(+), 13 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index c6d8a02ecd56..1e37ee6ae0ba 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -256,7 +256,7 @@ void hugetlb_add_anon_rmap(struct folio *, struct vm_area_struct *, void hugetlb_add_new_anon_rmap(struct folio *, struct vm_area_struct *, unsigned long address); -/* See page_try_dup_anon_rmap() */ +/* See folio_try_dup_anon_rmap_*() */ static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, struct vm_area_struct *vma) { @@ -481,16 +481,6 @@ static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio, #endif } -static inline int page_try_dup_anon_rmap(struct page *page, bool compound, - struct vm_area_struct *vma) -{ - struct folio *folio = page_folio(page); - - if (likely(!compound)) - return folio_try_dup_anon_rmap_pte(folio, page, vma); - return folio_try_dup_anon_rmap_pmd(folio, page, vma); -} - /** * page_try_share_anon_rmap - try marking an exclusive anonymous page possibly * shared to prepare for KSM or temporary unmapping @@ -499,8 +489,8 @@ static inline int page_try_dup_anon_rmap(struct page *page, bool compound, * The caller needs to hold the PT lock and has to have the page table entry * cleared/invalidated. * - * This is similar to page_try_dup_anon_rmap(), however, not used during fork() - * to duplicate a mapping, but instead to prepare for KSM or temporarily + * This is similar to folio_try_dup_anon_rmap_*(), however, not used during + * fork() to duplicate a mapping, but instead to prepare for KSM or temporarily * unmapping a page (swap, migration) via folio_remove_rmap_*(). * * Marking the page shared can only fail if the page may be pinned; device From patchwork Mon Dec 11 15:56:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A6B7C4167B for ; Mon, 11 Dec 2023 15:58:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 014ED6B0140; Mon, 11 Dec 2023 10:58:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F06726B0142; Mon, 11 Dec 2023 10:58:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D586E6B0143; Mon, 11 Dec 2023 10:58:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id BF0636B0140 for ; Mon, 11 Dec 2023 10:58:38 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 95E3C1A072C for ; Mon, 11 Dec 2023 15:58:38 +0000 (UTC) X-FDA: 81554995116.06.DFC36CB Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf04.hostedemail.com (Postfix) with ESMTP id D6D7240013 for ; Mon, 11 Dec 2023 15:58:36 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XlnTavQR; spf=pass (imf04.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310316; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lJSSYac1nr+7xVv+4BgklrjvdDaeoEoSEBfa7pAELWg=; b=59z3v5Q+P8bHnmEUY/TM51C75lwW3D8NjjskaNiVET7RXE2rkYxig3i51ntxRmaLGoCNif JkzSKIZxNmmW+tCdPbXP14spKkpa/4aEARrJJ+At4yYwht91VJUy3LEnl3PQTcHASN7PZX sUYr6SKweiLpM7I+5SYtO7RaitA+WHI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310316; a=rsa-sha256; cv=none; b=ZD3g3wbvqt8VoZZHGh+aBUwLz9uxG0FZAZqYNmvPYPiqm7JcOREjKyO0O8mK1APRlpwtYf 7/wOl6VK8/6fTg+im5IlLEh73lI3iJI5El//880Sd6lxmVT9VPkaFoE7jQ1GcmEs8CQ2Jg JnKHKZAJ1OwrfvtrH1JCaLH6di9RNY0= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XlnTavQR; spf=pass (imf04.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310316; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lJSSYac1nr+7xVv+4BgklrjvdDaeoEoSEBfa7pAELWg=; b=XlnTavQRqdWzVj+4MvPJbXjjkfsqJsfcZ9KJZ5tlHG4bAO0IvMcRu1UmGD6gnNnocmRCrA c8R0o0pZCAQvuEdEswjffvq7SqzOVjt7c/3YOCg/JoKCiI1VHPzL7YnRrUkuSEDvryn0P1 5fwpR6kMUYx1unkPSmO1zs/xUnkibvE= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-616-cwKjNob0OqmmW5y0fjpgYA-1; Mon, 11 Dec 2023 10:58:33 -0500 X-MC-Unique: cwKjNob0OqmmW5y0fjpgYA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id B6830185A787; Mon, 11 Dec 2023 15:58:32 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id 14FA71121306; Mon, 11 Dec 2023 15:58:30 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 38/39] mm: convert page_try_share_anon_rmap() to folio_try_share_anon_rmap_[pte|pmd]() Date: Mon, 11 Dec 2023 16:56:51 +0100 Message-ID: <20231211155652.131054-39-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: D6D7240013 X-Rspam-User: X-Stat-Signature: mq979hfzzfh7qx1gydnqxpxqgi4sf7fu X-Rspamd-Server: rspam03 X-HE-Tag: 1702310316-158253 X-HE-Meta: U2FsdGVkX19RFQYL9kENbyH+11gk5F6Za/NxWWPA/0lgJk6GwKI3sTEM48ZaFy79p05dEpytQTmKwgLPQCpUgGjOeY1GDkDkA6ekPat85bz8eMotKLix3NZBaWlQK72eizh5qDJ0HW5vGSZT56n/nw234zU1KBknb/Ql6++WVcOHEhnIU1AwHi0uQMIkLkSdQlkBGODZPBwztksiT72Yr+RXAMpq2AarJoTJup3vBsHB7g2RmRmnPBtdkLj/oJZhpgxyu/toZXGmW6Y1XcQnfayTSpOaWCw2tPqVO4H2zonSmicqfsMcxUvQVHNmU2UFCwTZ9L8MAbL6EVUvOgYJ7DWE/95wf//Ur+aXczIIhwxtiIf4IZU/56ZSthqooq5j/7pXN+0qebhEbVV56KFnMh+y5eRul5/QZOJ8HwaF6TZLezrpWTPTfW9NDi9TAgljklLOATnTgwZ/soIjXtMVBOexBO332tlxiWP4U+3RO7MDqwvHyjxB1TPJdcrsBG1WBoa/DynzehGXm0R1CNJcfZKzGaDnu1v5+fsUpxmwyTQ801Fpg0pbi6yRyvmEIL0605YM1sYc/quo2zSAl6RZIXQcXAs5ceJJCuJW3IojwaV3YvBekBhKuf20Tlh4uY2pmyG5Org4X4c/NRLXHwoqi2jKNVLsvKjUTXFXpvezPkk+Obhxr40HDgDRhuCIfbkUQwEKbiAiwr8sLCi06kEpqIkKuwMcGZudP3COc23drHURojLyF/3SnLAm/ST3TEVHCcC9eZAOPj3NMa5LIaEZCTVUbpcORvZ2LPMQtAqi6NuG4rTaOWRFPbBz8qkUiiRxmSQqvonO7QYVqaDg5NMZxGT4sCX/Bu7vcr8FJWe3wZhlOXqUzK64Oa0k7a0HSs5a2BWquqPtZB8cMPfr8EY2DJf/Ke/JHw5p+VQ76XpzC95FJrwHKsSx+mDw1WGzXPQvP6H3b1pCLluf5m6AC8J esNTiC+9 SBoGdFIg7EQxdfgggVYFtJrfc0Ay9+C22f0+68Z+NnkX7enWO91oyLJM7I4btybZMdp62bh14gu+hvjkGMYfnMT/JLKJ1kffmJcWoeifyck/DC3wuvh3tvgRX/9a+D9Oj/bZp24SWxopsQ77q0Cn09jEDjpSFW8ctVoPl14DwX0OYCH25rspdi+oCCqWzJ30uRSQcrJVLxeVGfdb200PTP62SN58PySGTGsn6 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert it like we converted all the other rmap functions. Don't introduce folio_try_share_anon_rmap_ptes() for now, as we don't have a user that wants rmap batching in sight. Pretty easy to add later. All users are easy to convert -- only ksm.c doesn't use folios yet but that is left for future work -- so let's just do it in a single shot. While at it, turn the BUG_ON into a WARN_ON_ONCE. Note that page_try_share_anon_rmap() so far didn't care about pte/pmd mappings (no compound parameter). We're changing that so we can perform better sanity checks and make the code actually more readable/consistent. For example, __folio_rmap_sanity_checks() will make sure that a PMD range actually falls completely into the folio. Signed-off-by: David Hildenbrand --- include/linux/rmap.h | 95 +++++++++++++++++++++++++++++++++----------- mm/gup.c | 2 +- mm/huge_memory.c | 9 +++-- mm/internal.h | 4 +- mm/ksm.c | 5 ++- mm/migrate_device.c | 2 +- mm/rmap.c | 11 ++--- 7 files changed, 89 insertions(+), 39 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 1e37ee6ae0ba..1e54a28cc884 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -272,7 +272,7 @@ static inline int hugetlb_try_dup_anon_rmap(struct folio *folio, return 0; } -/* See page_try_share_anon_rmap() */ +/* See folio_try_share_anon_rmap_*() */ static inline int hugetlb_try_share_anon_rmap(struct folio *folio) { VM_WARN_ON_FOLIO(!folio_test_hugetlb(folio), folio); @@ -481,30 +481,15 @@ static inline int folio_try_dup_anon_rmap_pmd(struct folio *folio, #endif } -/** - * page_try_share_anon_rmap - try marking an exclusive anonymous page possibly - * shared to prepare for KSM or temporary unmapping - * @page: the exclusive anonymous page to try marking possibly shared - * - * The caller needs to hold the PT lock and has to have the page table entry - * cleared/invalidated. - * - * This is similar to folio_try_dup_anon_rmap_*(), however, not used during - * fork() to duplicate a mapping, but instead to prepare for KSM or temporarily - * unmapping a page (swap, migration) via folio_remove_rmap_*(). - * - * Marking the page shared can only fail if the page may be pinned; device - * private pages cannot get pinned and consequently this function cannot fail. - * - * Returns 0 if marking the page possibly shared succeeded. Returns -EBUSY - * otherwise. - */ -static inline int page_try_share_anon_rmap(struct page *page) +static __always_inline int __folio_try_share_anon_rmap(struct folio *folio, + struct page *page, int nr_pages, enum rmap_mode mode) { - VM_BUG_ON_PAGE(!PageAnon(page) || !PageAnonExclusive(page), page); + VM_WARN_ON_FOLIO(!folio_test_anon(folio), folio); + VM_WARN_ON_FOLIO(!PageAnonExclusive(page), folio); + __folio_rmap_sanity_checks(folio, page, nr_pages, mode); - /* device private pages cannot get pinned via GUP. */ - if (unlikely(is_device_private_page(page))) { + /* device private folios cannot get pinned via GUP. */ + if (unlikely(folio_is_device_private(folio))) { ClearPageAnonExclusive(page); return 0; } @@ -555,7 +540,7 @@ static inline int page_try_share_anon_rmap(struct page *page) if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) smp_mb(); - if (unlikely(page_maybe_dma_pinned(page))) + if (unlikely(folio_maybe_dma_pinned(folio))) return -EBUSY; ClearPageAnonExclusive(page); @@ -568,6 +553,68 @@ static inline int page_try_share_anon_rmap(struct page *page) return 0; } +/** + * folio_try_share_anon_rmap_pte - try marking an exclusive anonymous page + * mapped by a PTE possibly shared to prepare + * for KSM or temporary unmapping + * @folio: The folio to share a mapping of + * @page: The mapped exclusive page + * + * The caller needs to hold the page table lock and has to have the page table + * entries cleared/invalidated. + * + * This is similar to folio_try_dup_anon_rmap_pte(), however, not used during + * fork() to duplicate mappings, but instead to prepare for KSM or temporarily + * unmapping parts of a folio (swap, migration) via folio_remove_rmap_pte(). + * + * Marking the mapped page shared can only fail if the folio maybe pinned; + * device private folios cannot get pinned and consequently this function cannot + * fail. + * + * Returns 0 if marking the mapped page possibly shared succeeded. Returns + * -EBUSY otherwise. + */ +static inline int folio_try_share_anon_rmap_pte(struct folio *folio, + struct page *page) +{ + return __folio_try_share_anon_rmap(folio, page, 1, RMAP_MODE_PTE); +} + +/** + * folio_try_share_anon_rmap_pmd - try marking an exclusive anonymous page + * range mapped by a PMD possibly shared to + * prepare for temporary unmapping + * @folio: The folio to share the mapping of + * @page: The first page to share the mapping of + * + * The page range of the folio is defined by [page, page + HPAGE_PMD_NR) + * + * The caller needs to hold the page table lock and has to have the page table + * entries cleared/invalidated. + * + * This is similar to folio_try_dup_anon_rmap_pmd(), however, not used during + * fork() to duplicate a mapping, but instead to prepare for temporarily + * unmapping parts of a folio (swap, migration) via folio_remove_rmap_pmd(). + * + * Marking the mapped pages shared can only fail if the folio maybe pinned; + * device private folios cannot get pinned and consequently this function cannot + * fail. + * + * Returns 0 if marking the mapped pages possibly shared succeeded. Returns + * -EBUSY otherwise. + */ +static inline int folio_try_share_anon_rmap_pmd(struct folio *folio, + struct page *page) +{ +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + return __folio_try_share_anon_rmap(folio, page, HPAGE_PMD_NR, + RMAP_MODE_PMD); +#else + WARN_ON_ONCE(true); + return -EBUSY; +#endif +} + /* * Called from mm/vmscan.c to handle paging out */ diff --git a/mm/gup.c b/mm/gup.c index 0a5f0e91bfec..df83182ec72d 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -177,7 +177,7 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags) /* * Adjust the pincount before re-checking the PTE for changes. * This is essentially a smp_mb() and is paired with a memory - * barrier in page_try_share_anon_rmap(). + * barrier in folio_try_share_anon_rmap_*(). */ smp_mb__after_atomic(); diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 34f878916621..c681296fa429 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2523,10 +2523,11 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, * In case we cannot clear PageAnonExclusive(), split the PMD * only and let try_to_migrate_one() fail later. * - * See page_try_share_anon_rmap(): invalidate PMD first. + * See folio_try_share_anon_rmap_pmd(): invalidate PMD first. */ anon_exclusive = PageAnonExclusive(page); - if (freeze && anon_exclusive && page_try_share_anon_rmap(page)) + if (freeze && anon_exclusive && + folio_try_share_anon_rmap_pmd(folio, page)) freeze = false; if (!freeze) { rmap_t rmap_flags = RMAP_NONE; @@ -3554,9 +3555,9 @@ int set_pmd_migration_entry(struct page_vma_mapped_walk *pvmw, flush_cache_range(vma, address, address + HPAGE_PMD_SIZE); pmdval = pmdp_invalidate(vma, address, pvmw->pmd); - /* See page_try_share_anon_rmap(): invalidate PMD first. */ + /* See folio_try_share_anon_rmap_pmd(): invalidate PMD first. */ anon_exclusive = folio_test_anon(folio) && PageAnonExclusive(page); - if (anon_exclusive && page_try_share_anon_rmap(page)) { + if (anon_exclusive && folio_try_share_anon_rmap_pmd(folio, page)) { set_pmd_at(mm, address, pvmw->pmd, pmdval); return -EBUSY; } diff --git a/mm/internal.h b/mm/internal.h index a94355e70bd7..29589bc3f046 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1047,7 +1047,7 @@ enum { * * Ordinary GUP: Using the PT lock * * GUP-fast and fork(): mm->write_protect_seq * * GUP-fast and KSM or temporary unmapping (swap, migration): see - * page_try_share_anon_rmap() + * folio_try_share_anon_rmap_*() * * Must be called with the (sub)page that's actually referenced via the * page table entry, which might not necessarily be the head page for a @@ -1090,7 +1090,7 @@ static inline bool gup_must_unshare(struct vm_area_struct *vma, return is_cow_mapping(vma->vm_flags); } - /* Paired with a memory barrier in page_try_share_anon_rmap(). */ + /* Paired with a memory barrier in folio_try_share_anon_rmap_*(). */ if (IS_ENABLED(CONFIG_HAVE_FAST_GUP)) smp_rmb(); diff --git a/mm/ksm.c b/mm/ksm.c index b3d0cfaa2533..be76e9dabf4f 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1161,8 +1161,9 @@ static int write_protect_page(struct vm_area_struct *vma, struct page *page, goto out_unlock; } - /* See page_try_share_anon_rmap(): clear PTE first. */ - if (anon_exclusive && page_try_share_anon_rmap(page)) { + /* See folio_try_share_anon_rmap_pte(): clear PTE first. */ + if (anon_exclusive && + folio_try_share_anon_rmap_pte(page_folio(page), page)) { set_pte_at(mm, pvmw.address, pvmw.pte, entry); goto out_unlock; } diff --git a/mm/migrate_device.c b/mm/migrate_device.c index c51c99151ebb..9d0c1ad73722 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -202,7 +202,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, if (anon_exclusive) { pte = ptep_clear_flush(vma, addr, ptep); - if (page_try_share_anon_rmap(page)) { + if (folio_try_share_anon_rmap_pte(folio, page)) { set_pte_at(mm, addr, ptep, pte); folio_unlock(folio); folio_put(folio); diff --git a/mm/rmap.c b/mm/rmap.c index b08dd7d6779d..45296739236f 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1868,9 +1868,9 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, break; } - /* See page_try_share_anon_rmap(): clear PTE first. */ + /* See folio_try_share_anon_rmap(): clear PTE first. */ if (anon_exclusive && - page_try_share_anon_rmap(subpage)) { + folio_try_share_anon_rmap_pte(folio, subpage)) { swap_free(entry); set_pte_at(mm, address, pvmw.pte, pteval); ret = false; @@ -2144,7 +2144,8 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, pte_t swp_pte; if (anon_exclusive) - BUG_ON(page_try_share_anon_rmap(subpage)); + WARN_ON_ONCE(folio_try_share_anon_rmap_pte(folio, + subpage)); /* * Store the pfn of the page in a special migration @@ -2215,7 +2216,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, VM_BUG_ON_PAGE(pte_write(pteval) && folio_test_anon(folio) && !anon_exclusive, subpage); - /* See page_try_share_anon_rmap(): clear PTE first. */ + /* See folio_try_share_anon_rmap_pte(): clear PTE first. */ if (folio_test_hugetlb(folio)) { if (anon_exclusive && hugetlb_try_share_anon_rmap(folio)) { @@ -2226,7 +2227,7 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, break; } } else if (anon_exclusive && - page_try_share_anon_rmap(subpage)) { + folio_try_share_anon_rmap_pte(folio, subpage)) { set_pte_at(mm, address, pvmw.pte, pteval); ret = false; page_vma_mapped_walk_done(&pvmw); From patchwork Mon Dec 11 15:56:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13487479 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8FD5C4167B for ; Mon, 11 Dec 2023 15:58:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 62E3C6B0144; Mon, 11 Dec 2023 10:58:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5B60E6B0146; Mon, 11 Dec 2023 10:58:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 430C96B0147; Mon, 11 Dec 2023 10:58:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 309396B0144 for ; Mon, 11 Dec 2023 10:58:42 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 02EA040762 for ; Mon, 11 Dec 2023 15:58:41 +0000 (UTC) X-FDA: 81554995284.04.F51EFB7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf30.hostedemail.com (Postfix) with ESMTP id 338A580022 for ; Mon, 11 Dec 2023 15:58:39 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=g2OabzK2; spf=pass (imf30.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1702310320; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8kCOKQnn/YILhh4/y6F4VVgOwWNItKHStdMCkjwZ7+g=; b=ASxIn/KSBt7BDQlzYDjVh10FwbMVgwZnItGuo3+8cTiZX1tir3Y2u8QAQ4hvE/tqL/NRB5 hQ/v/ZXDyMrRt07eMKU1UtSkzrpr/5NTicJfaxCJyYVI63XjZhnMbJgVBYFisM/5KQEF7H BrlqCLJBByTPCctW3InLekDue2L/ki4= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=g2OabzK2; spf=pass (imf30.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1702310320; a=rsa-sha256; cv=none; b=cGGXBbKxxrft+jWomrxX1Yat4QdBPvEbek9bYNa7Zn9ZTx8biS+qWq44wcqYmOxCh0ti82 ohh98p1ncpeWxuVzly6ibIryvMJsXYRFkjsFRWnZZknoEefCoUqKhZrWMW9GpqxWAvP1eG v17CzKriSI7Bp/LUirqydFRp61ZtHE4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1702310319; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8kCOKQnn/YILhh4/y6F4VVgOwWNItKHStdMCkjwZ7+g=; b=g2OabzK2ErmylqZsX4QjOHnqqRqbcp61bt+5PhpX7WKdKRXhR62vrv/L7iEY8W6nrIFM4/ AWY0XjhWTMHXcqb6ZtT9n1wK3Y9wbb6Cg4s64ziYmPSaPD5HRRdoiQy9smVBAYeD0Cotly XEpylHuvathX5l0lnigfdQeUGyysFa4= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-461-PqgcZlcINPWhAdvqUvsCRQ-1; Mon, 11 Dec 2023 10:58:35 -0500 X-MC-Unique: PqgcZlcINPWhAdvqUvsCRQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id E3F70848A68; Mon, 11 Dec 2023 15:58:34 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.192.166]) by smtp.corp.redhat.com (Postfix) with ESMTP id F28F81121306; Mon, 11 Dec 2023 15:58:32 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , "Matthew Wilcox (Oracle)" , Hugh Dickins , Ryan Roberts , Yin Fengwei , Mike Kravetz , Muchun Song , Peter Xu Subject: [PATCH v1 39/39] mm/rmap: rename COMPOUND_MAPPED to ENTIRELY_MAPPED Date: Mon, 11 Dec 2023 16:56:52 +0100 Message-ID: <20231211155652.131054-40-david@redhat.com> In-Reply-To: <20231211155652.131054-1-david@redhat.com> References: <20231211155652.131054-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Queue-Id: 338A580022 X-Rspam-User: X-Stat-Signature: iorhmyfhsgr39dokgtthh91z3r5nuso1 X-Rspamd-Server: rspam01 X-HE-Tag: 1702310319-326540 X-HE-Meta: U2FsdGVkX18h9IdFzOB8HHPhR38lF8g2rBb9gpD8HYHXd6s3svc6qHLzLsFanPX+47IYfkxZHEeKKM72L/98gPlJrI8k27hxXJlX1AjmgsvZCtKmuwV0E3fLzvBBxasjyqB8Mj6FfyxkotLlhewYpGTBgdzB9MxB5M9brWYMBVrjBOvnx6V6LmkYmBmxjb6eNvOZB+ydMeCy7fdZRyCdRek0gtV5Jc9yF89P7q9ZSFetlU0OocDfz4LNfyf+Yp3MeqbAj43ySvRFRNeDZ/sacr1RLMSNTlf1zoik4CLu5gKkvxhwkf1K7ThRpf2BLT7ELZ0ONuuW53l6QA2PIN9FE681nsz4AV0eVNN+Cwn0M1M9JiXxii0REzpKNd756olnDfTuuZ/nqyrmZFh0M/UaVFn5Dap9gAwLLbr4BcKzxiEe8FjZ9dZQ6t3ywFr7HHUj4iX/CflNcj3xgFs/55rld3o/soQLPktWHomZHGat25FVaDAtH9/1y2GylLJQuWd7ezbB4vB1HbyTU7Jj4hQXC1v/+m3taDqcD6l1LYUA+MGk0JcyGqRwy055m8P2fydP+Cab/4OP5gsHGPoLioFZ3O1c99S6a1eHu5HhndWwJrzIYCxXblK+Bq7QufuxOjwNk8Y/OAm/1/O4xzXemmo33MZWG8aZBmo4cIkI4hId2rhv0EaMSNCMrdeZx04XtgMMhCZ2MRbCzngQg03aqfEMc4EcBlrqLf2Ozh3fA0iERsYZ2eahm2CBuQRXhiYtGR+doTofgXskS2CJOB/QeGS62Vj97Kcw2PzqAsus9CTHikN1O5lU51Q9xTnSxvK/YOlFuQ2oA7KLNM6AKK9rRcAACdv3sqAyut+AriOYYfr0TPpiIVpnGmJ6VMyew1tMcjnil72+ungaP/HFlmVxSoxpQ+3YrVnjje5sheJrUbWlQCsLGSP6CxtvwbEPs1vKg7clfq2qCZcpK73rX7cA6it /nQBUK2R avVm2ND7Y/HWqOcMiiSmaiX337JahbT67Ywtm8LdZ5FcmXRFF1NBveX+MBszLgiPiw8xOy9T+NQY4FPHQTz28cukewgtUUNWoy0owp1ngtYIsEWLRxfyxYovzGRibgnAAvlOmO9GrSlwKgFlN1pe0FMeGquv+mGIrn1XqTJoipYKOsisQ6//p364LZ8Vke+MLFi9t7CaD3uaRYDATYsRxq4DgS50zh+Rg4n5v X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We removed all "bool compound" and RMAP_COMPOUND parameters. Let's remove the remaining "compound" terminology by making COMPOUND_MAPPED match the "folio->_entire_mapcount" terminology, renaming it to ENTIRELY_MAPPED. ENTIRELY_MAPPED is only used when the whole folio is mapped using a single page table entry (e.g., a single PMD mapping a PMD-sized THP). For now, we don't support mapping any THP bigger than that, so ENTIRELY_MAPPED only applies to PMD-mapped PMD-sized THP only. Signed-off-by: David Hildenbrand --- Documentation/mm/transhuge.rst | 2 +- mm/internal.h | 6 +++--- mm/rmap.c | 18 +++++++++--------- 3 files changed, 13 insertions(+), 13 deletions(-) diff --git a/Documentation/mm/transhuge.rst b/Documentation/mm/transhuge.rst index cf81272a6b8b..93c9239b9ebe 100644 --- a/Documentation/mm/transhuge.rst +++ b/Documentation/mm/transhuge.rst @@ -117,7 +117,7 @@ pages: - map/unmap of a PMD entry for the whole THP increment/decrement folio->_entire_mapcount and also increment/decrement - folio->_nr_pages_mapped by COMPOUND_MAPPED when _entire_mapcount + folio->_nr_pages_mapped by ENTIRELY_MAPPED when _entire_mapcount goes from -1 to 0 or 0 to -1. - map/unmap of individual pages with PTE entry increment/decrement diff --git a/mm/internal.h b/mm/internal.h index 29589bc3f046..188807d2aebc 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -54,12 +54,12 @@ void page_writeback_init(void); /* * If a 16GB hugetlb folio were mapped by PTEs of all of its 4kB pages, - * its nr_pages_mapped would be 0x400000: choose the COMPOUND_MAPPED bit + * its nr_pages_mapped would be 0x400000: choose the ENTIRELY_MAPPED bit * above that range, instead of 2*(PMD_SIZE/PAGE_SIZE). Hugetlb currently * leaves nr_pages_mapped at 0, but avoid surprise if it participates later. */ -#define COMPOUND_MAPPED 0x800000 -#define FOLIO_PAGES_MAPPED (COMPOUND_MAPPED - 1) +#define ENTIRELY_MAPPED 0x800000 +#define FOLIO_PAGES_MAPPED (ENTIRELY_MAPPED - 1) /* * Flags passed to __show_mem() and show_free_areas() to suppress output in diff --git a/mm/rmap.c b/mm/rmap.c index 45296739236f..53753834a10d 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1173,7 +1173,7 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, first = atomic_inc_and_test(&page->_mapcount); if (first && folio_test_large(folio)) { first = atomic_inc_return_relaxed(mapped); - first = (first < COMPOUND_MAPPED); + first = (first < ENTIRELY_MAPPED); } if (first) @@ -1183,15 +1183,15 @@ static __always_inline unsigned int __folio_add_rmap(struct folio *folio, case RMAP_MODE_PMD: first = atomic_inc_and_test(&folio->_entire_mapcount); if (first) { - nr = atomic_add_return_relaxed(COMPOUND_MAPPED, mapped); - if (likely(nr < COMPOUND_MAPPED + COMPOUND_MAPPED)) { + nr = atomic_add_return_relaxed(ENTIRELY_MAPPED, mapped); + if (likely(nr < ENTIRELY_MAPPED + ENTIRELY_MAPPED)) { *nr_pmdmapped = folio_nr_pages(folio); nr = *nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of a remove and another add? */ if (unlikely(nr < 0)) nr = 0; } else { - /* Raced ahead of a remove of COMPOUND_MAPPED */ + /* Raced ahead of a remove of ENTIRELY_MAPPED */ nr = 0; } } @@ -1434,7 +1434,7 @@ void folio_add_new_anon_rmap(struct folio *folio, struct vm_area_struct *vma, } else { /* increment count (starts at -1) */ atomic_set(&folio->_entire_mapcount, 0); - atomic_set(&folio->_nr_pages_mapped, COMPOUND_MAPPED); + atomic_set(&folio->_nr_pages_mapped, ENTIRELY_MAPPED); SetPageAnonExclusive(&folio->page); __lruvec_stat_mod_folio(folio, NR_ANON_THPS, nr); } @@ -1516,7 +1516,7 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, last = atomic_add_negative(-1, &page->_mapcount); if (last && folio_test_large(folio)) { last = atomic_dec_return_relaxed(mapped); - last = (last < COMPOUND_MAPPED); + last = (last < ENTIRELY_MAPPED); } if (last) @@ -1526,15 +1526,15 @@ static __always_inline void __folio_remove_rmap(struct folio *folio, case RMAP_MODE_PMD: last = atomic_add_negative(-1, &folio->_entire_mapcount); if (last) { - nr = atomic_sub_return_relaxed(COMPOUND_MAPPED, mapped); - if (likely(nr < COMPOUND_MAPPED)) { + nr = atomic_sub_return_relaxed(ENTIRELY_MAPPED, mapped); + if (likely(nr < ENTIRELY_MAPPED)) { nr_pmdmapped = folio_nr_pages(folio); nr = nr_pmdmapped - (nr & FOLIO_PAGES_MAPPED); /* Raced ahead of another remove and an add? */ if (unlikely(nr < 0)) nr = 0; } else { - /* An add of COMPOUND_MAPPED raced ahead */ + /* An add of ENTIRELY_MAPPED raced ahead */ nr = 0; } }