From patchwork Mon Oct 2 14:29:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13406285 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF79AE784B3 for ; Mon, 2 Oct 2023 14:30:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 69DFC6B0140; Mon, 2 Oct 2023 10:30:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 64DE66B0158; Mon, 2 Oct 2023 10:30:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 53D256B0159; Mon, 2 Oct 2023 10:30:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 40B356B0140 for ; Mon, 2 Oct 2023 10:30:01 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 092DF402DD for ; Mon, 2 Oct 2023 14:30:01 +0000 (UTC) X-FDA: 81300755802.30.165D64D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 284594000D for ; Mon, 2 Oct 2023 14:29:57 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Y5U0Y6mU; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696256998; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=74/4kguVt+JNeClEzh2bgjVFXMUXyl4kPsimySG+lfk=; b=4nlyeFJlFgZJqIfmN+hpUo/B7tNcLdoSz1PAvYyxyNMvn+zYERGeAEGqkgPMAKYX5r1ewV LeOGdvmOq4oeWNXjfi7SBoIVd8/hMCqGiG19UyhtPAzuIEZX02NCb7wQuzslmxNbYKRT2R t2fANdy0Pd55pnICv5qYd3bURZtibcw= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Y5U0Y6mU; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696256998; a=rsa-sha256; cv=none; b=FPsWYnxqQdW6Ucu5EVl8BcgamK8VtmhVknTj3Yrksemaue9VDML9McwErXUzCyTp+xLZcf O8znbqv+1OdIyca1XfQfpLFbnAn1DHD6M97PPD1hyf9HcPZM9ylgatP5qsQiumi1ALTZhS eBXG8QwAuLtpX47eiFG2emeYjmUXOZU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1696256997; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=74/4kguVt+JNeClEzh2bgjVFXMUXyl4kPsimySG+lfk=; b=Y5U0Y6mUvc8tOXPyv+L+x1UMw0c1OVT8QsrdyGbyM1M/50aKRRxm6XtRpswBuwazG1AzP7 lV8tLpNd/idC0q8/4mPNDBCJTLSbwbVei6gZ1VJjwg5HYTXSeCmwzEfP0Cw7hYv5EZpVI8 pIkD6uJfBxaJmPygfwPQdY3+Cd/aZmA= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-237-MXePQoblNP-G9nwx88mC8A-1; Mon, 02 Oct 2023 10:29:53 -0400 X-MC-Unique: MXePQoblNP-G9nwx88mC8A-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id C5A84801779; Mon, 2 Oct 2023 14:29:52 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.194.19]) by smtp.corp.redhat.com (Postfix) with ESMTP id A52811005B90; Mon, 2 Oct 2023 14:29:51 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Mike Kravetz , Muchun Song , Suren Baghdasaryan Subject: [PATCH v1 1/3] mm/rmap: move SetPageAnonExclusive() out of page_move_anon_rmap() Date: Mon, 2 Oct 2023 16:29:47 +0200 Message-ID: <20231002142949.235104-2-david@redhat.com> In-Reply-To: <20231002142949.235104-1-david@redhat.com> References: <20231002142949.235104-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspamd-Queue-Id: 284594000D X-Rspam-User: X-Stat-Signature: f1qz6sidbxhczrb1jw4s5rppmi95jg3g X-Rspamd-Server: rspam01 X-HE-Tag: 1696256997-845418 X-HE-Meta: U2FsdGVkX1+PFMVT7g83O3OLDieWyicCBPVzgAzvUHjuTLm0KdfYoVjxFxHSrd53+tOB8dNc8FgptLzkG924/olzK5jXMGOKzkEPEIyIxyVW3Ns/jZf3oc/snlfS6TUR43xfaEwEGN0ZFnsu6NJTndcgOe2Bzxb2LampT08ttSt5p4+U5ffMEjjsAVgSeyz5VAD2jwFFgj9ahBTx6VN0eCNMSQC1ecbrG+fV00B9GtdDru5NrhJcfv/0JfEMQT/ppNvejIQ9fVkeCQ8iw7FlhUugyBYEaizKjsSZ+fPon6EuSrAmnXNrHMkIdx5DgXuYqtvi3rRmjlA9QY55ePLdJnncO2m1tcpY/6mV5ERcByOxYBP/x/5FjhhlUjQOfLTCNaKnorzLj6EhTrpybCg+6P75zEILCGOQZ7kvCmE2gycLFwHXyAA5xCZuwMTi90Jc/cGgCYN5Uo/lqkIyqXx5uszz+iPhg/eDPC6OPmEOPov6dm4bjpgOz/1mR75rZRw7RN8IAQFbCjGIVITEMDtKdJuIvBaP4P5gDy0QFAdQQjsYLqb5p0GbLdM/IhCCq+aAYRcNieFsXogaxanhYuxzNVpGlIC0V/aF7Lv4BMlrO45tUpHuyMXOQXjAxcSdz5D1LQa69lq0YjcymonrCceH6pylvUuuLiKjlJRLQbXTkewWLdIetO+7GsIXGhKg6KGaZe3mUM4oPSMrBsBdZs+JMlmiRi1TblVTyP81BsexbsNpsxSfeIGeNO5qht8T/GeEtK4o0h92yTvSv0JNQp2oPHofZ/IRuFQg5pI8sPuQn9Urcjs6ZtTzhoyFi7S3G0Y1sxShf+07pDGQIODFkgyJK+KoNS0a+ib7oMd+iE3jRAJnl/0AWAp0mpTaaee/k4xLZA1jyEb27D1lLQ1Ibc734kHr1NpKa7HMeOcXnxIap039qGGZibY9ca4cyw9ASWJJucAlAHTGu9/qRAzHlP9 RyOIzf2t 7JDea7xYoojQ/1pQpfSYV8uxk0zS1G1gRMlZMuKFNfd5IdzJNihWvv8Xzt3BsuDV+ZQ3H5uGqsJjSInSgLjV3BQt2mAnGhYvNlBvIbmDxrnyFW1rG0O63pGdguSPn/VLZS1MQO0KQEuPQ+RE6PlFviXIpoEHmnSy3CD8trn2dIT1+oNoFqoNSTyGgBhNO6gDODcdSjTELl+M8cc6dvbuU6FEDlA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Let's move it into the caller: there is a difference between whether an anon folio can only be mapped by one process (e.g., into one VMA), and whether it is truly exclusive (e.g., no references -- including GUP -- from other processes). Further, for large folios the page might not actually be pointing at the head page of the folio, so it better be handled in the caller. This is a preparation for converting page_move_anon_rmap() to consume a folio. Signed-off-by: David Hildenbrand Reviewed-by: Suren Baghdasaryan Reviewed-by: Vishal Moola (Oracle) --- mm/huge_memory.c | 1 + mm/hugetlb.c | 4 +++- mm/memory.c | 1 + mm/rmap.c | 1 - 4 files changed, 5 insertions(+), 2 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index e54fb9c542bb..01d0d65ece13 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1506,6 +1506,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf) pmd_t entry; page_move_anon_rmap(page, vma); + SetPageAnonExclusive(page); folio_unlock(folio); reuse: if (unlikely(unshare)) { diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 9c22297d9c57..24591fc145ff 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5460,8 +5460,10 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, * owner and can reuse this page. */ if (folio_mapcount(old_folio) == 1 && folio_test_anon(old_folio)) { - if (!PageAnonExclusive(&old_folio->page)) + if (!PageAnonExclusive(&old_folio->page)) { page_move_anon_rmap(&old_folio->page, vma); + SetPageAnonExclusive(&old_folio->page); + } if (likely(!unshare)) set_huge_ptep_writable(vma, haddr, ptep); diff --git a/mm/memory.c b/mm/memory.c index d4820802b01b..9de231c92769 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3484,6 +3484,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) * sunglasses. Hit it. */ page_move_anon_rmap(vmf->page, vma); + SetPageAnonExclusive(vmf->page); folio_unlock(folio); reuse: if (unlikely(unshare)) { diff --git a/mm/rmap.c b/mm/rmap.c index 77222adccda1..854ccbd66954 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1165,7 +1165,6 @@ void page_move_anon_rmap(struct page *page, struct vm_area_struct *vma) * folio_test_anon()) will not see one without the other. */ WRITE_ONCE(folio->mapping, anon_vma); - SetPageAnonExclusive(page); } /** From patchwork Mon Oct 2 14:29:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13406288 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 998B1E784B6 for ; Mon, 2 Oct 2023 14:30:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 84D6A8D0001; Mon, 2 Oct 2023 10:30:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7D57B6B015C; Mon, 2 Oct 2023 10:30:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 601A98D0001; Mon, 2 Oct 2023 10:30:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 44F616B015B for ; Mon, 2 Oct 2023 10:30:06 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id EFA23402EE for ; Mon, 2 Oct 2023 14:30:05 +0000 (UTC) X-FDA: 81300755970.14.17FB4D0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 0CBAF40020 for ; Mon, 2 Oct 2023 14:30:03 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SH4EzYcO; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696257004; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iuOLWxV07WMAFicgIRp5L49gcu/vw0Qb9ZJxOn72iWI=; b=KalD5pBRCx6tVGnkasZh1AK0w42m7xeQHgRRfssX2NyrRII3QfE8GMz3G8q+TQv6NOgmz7 APS6/3NYrnBOTZ70bbdyln7mPlFUVxKz0pb94OL3QPdC0bf8sEJYmg8CTETXoLRjK5UDFx s+2jaQsb91knLAomz3aEzKs4bRBwuIA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696257004; a=rsa-sha256; cv=none; b=N+w6t4EHQYgCjIz6xUWBIRpf2kpy+KdShGEb4zq9kXif0UZa8txQ4es5D0QruNXeoJJqzy mSIieP12jS4s/Wvskg6fwuorea7UaPNjnvepdAiva2emvlUvJvu8OJLpMMghzEnxvvbmW6 j67AROP3vFDPj8Dl7yRm80rTtU9inMM= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=SH4EzYcO; spf=pass (imf12.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1696257002; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iuOLWxV07WMAFicgIRp5L49gcu/vw0Qb9ZJxOn72iWI=; b=SH4EzYcOzBcGm+zVVvcPmPSVsILrgZUrkzhO1r+OZg1fkRDi50GHpFtbx+plBCV9/PmLE8 1Fyc/Ar92InS8IGosYydUXJb2mQSkTpuvI9Hcf0ZnhD9+mu7OyXIzVx7dXXytxJl5fCMk+ W9x2EUZ3vzbkd13HVZNXb2VgzjK1Bnk= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-564-A63WA2viPemAtiRYluLjVw-1; Mon, 02 Oct 2023 10:29:54 -0400 X-MC-Unique: A63WA2viPemAtiRYluLjVw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 26B71802C1A; Mon, 2 Oct 2023 14:29:54 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.194.19]) by smtp.corp.redhat.com (Postfix) with ESMTP id 09C2510EE6CE; Mon, 2 Oct 2023 14:29:52 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Mike Kravetz , Muchun Song , Suren Baghdasaryan Subject: [PATCH v1 2/3] mm/rmap: convert page_move_anon_rmap() to folio_move_anon_rmap() Date: Mon, 2 Oct 2023 16:29:48 +0200 Message-ID: <20231002142949.235104-3-david@redhat.com> In-Reply-To: <20231002142949.235104-1-david@redhat.com> References: <20231002142949.235104-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspamd-Queue-Id: 0CBAF40020 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 53s1hdhea64jminsfmkrb8dg1z1wgros X-HE-Tag: 1696257003-410225 X-HE-Meta: U2FsdGVkX18ni5bSyk7eia2+pOhSqLv1rRDEcYyznm90MOSzhwEtZ2VlGJiFb67mo4myvJzTTu3iXGKuTKixPsuzMej96Z1leTep+xc46kQBF122NiZXZ9qJacFJq71ccdhfsPhgKu3GvHt+sLLvWm4ulUX+TEwhi/081ZdPSfx2X5imtLuv3/fFpLUobqOOADncUtvpjyyPTFW8yTm+qMrU9YSrfZ7EQrnQTAQcxC4xeFYgtCwR+VQoGQciEHQrzXcSI8WhNGpeiA4/Uy7cCfEtd0di+sKOaZNjtjDm3JRHEJgG+7dqV1DQuluzRuyUF3I4i3vcQkv7+XMBgXMzMAigCV3oT7uW5gpMknjyTQ4n3x1YJKGng9+paSi3psu2hdCLGFquKkEvuLCxE50KCXG/tWi8ckJxL0kXD3u6Rd9cBgB+sQmLT+rkSSjaNVK0BPFR/d9IqVIG7OCML7xkfg3bKFediYqOSzZKgTEk/eZA7egnjwO4CqQU99iETi89te3TJJSUkxS1rzur3GbruP+10DzAC/hLUdyM1yDDmgbJgmPpprxrn12IB+46E+GVNcBHBm5HVP2VfKXdeX3/S2FMr/m5MEQBboJxEycP8tyibyaM2XBZ1HDuMJO3T9AOcEEEoA2I82x7qVaOYfPrrt3Q4gl+jgfeg8XUv3EeG7RA9dCLb5tMuycEqO1CBgrJE3jkQPMcf7Mt0j8JsK1nOJCgA9SfabRnVSlXhP08mGvHhgmZXANojA4yXDo7MuxCPbbJ7Ptq3TRt9N5Rz96dM3Rp4QYiqBEnXTVWjpQ5SD9fHIfqGC+I7+S6ZTKr41ZTV8mxhxMJ5cGPFP4oOUvR4wbDz3SrGPrOExC0BJwFO2oanbXNw7+qFX0WfFrHz3A/TydgF5E5/MaegQkgQGh87/SAGgm/NwJ3BqZ9GAgArnqG3zAldwH+A3ORh14/fxgs0Mi+WGNrEvAEanHpz1O cJgDBfqg LS9bUEdmIjzoabKKBcjv5K1+Ockfpu6Dh5G1UpRXuDLfWtaHnaHWneVAMzmPpaH7UDKXsR+pcf2V4TK15vdtnqepAyNGBWlzxD6PaPBEIxE7xMm7LtTLpaR288j+JJ5auM2kZBGABGk7kKdytBYGAAj+y0bg056fvardDTbZtL+0jY890TrZFaJsmsSHiMlyOfTkEMcEcurqiTqBaF4Hr6IdmXg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Let's convert it to consume a folio. Signed-off-by: David Hildenbrand Reviewed-by: Suren Baghdasaryan Reviewed-by: Vishal Moola (Oracle) --- include/linux/rmap.h | 2 +- mm/huge_memory.c | 2 +- mm/hugetlb.c | 2 +- mm/memory.c | 2 +- mm/rmap.c | 16 +++++++--------- 5 files changed, 11 insertions(+), 13 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 064b432a4033..8034eda972e5 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -194,7 +194,7 @@ typedef int __bitwise rmap_t; /* * rmap interfaces called when adding or removing pte of page */ -void page_move_anon_rmap(struct page *, struct vm_area_struct *); +void folio_move_anon_rmap(struct folio *, struct vm_area_struct *); void page_add_anon_rmap(struct page *, struct vm_area_struct *, unsigned long address, rmap_t flags); void page_add_new_anon_rmap(struct page *, struct vm_area_struct *, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 01d0d65ece13..08245226ccb8 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1505,7 +1505,7 @@ vm_fault_t do_huge_pmd_wp_page(struct vm_fault *vmf) if (folio_ref_count(folio) == 1) { pmd_t entry; - page_move_anon_rmap(page, vma); + folio_move_anon_rmap(folio, vma); SetPageAnonExclusive(page); folio_unlock(folio); reuse: diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 24591fc145ff..e52c6048e74f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5461,7 +5461,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, */ if (folio_mapcount(old_folio) == 1 && folio_test_anon(old_folio)) { if (!PageAnonExclusive(&old_folio->page)) { - page_move_anon_rmap(&old_folio->page, vma); + folio_move_anon_rmap(old_folio, vma); SetPageAnonExclusive(&old_folio->page); } if (likely(!unshare)) diff --git a/mm/memory.c b/mm/memory.c index 9de231c92769..1f0e3317cbdd 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3483,7 +3483,7 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) * and the folio is locked, it's dark out, and we're wearing * sunglasses. Hit it. */ - page_move_anon_rmap(vmf->page, vma); + folio_move_anon_rmap(folio, vma); SetPageAnonExclusive(vmf->page); folio_unlock(folio); reuse: diff --git a/mm/rmap.c b/mm/rmap.c index 854ccbd66954..37f05f33559b 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1141,19 +1141,17 @@ int folio_total_mapcount(struct folio *folio) } /** - * page_move_anon_rmap - move a page to our anon_vma - * @page: the page to move to our anon_vma - * @vma: the vma the page belongs to + * folio_move_anon_rmap - move a folio to our anon_vma + * @page: The folio to move to our anon_vma + * @vma: The vma the folio belongs to * - * When a page belongs exclusively to one process after a COW event, - * that page can be moved into the anon_vma that belongs to just that - * process, so the rmap code will not search the parent or sibling - * processes. + * When a folio belongs exclusively to one process after a COW event, + * that folio can be moved into the anon_vma that belongs to just that + * process, so the rmap code will not search the parent or sibling processes. */ -void page_move_anon_rmap(struct page *page, struct vm_area_struct *vma) +void folio_move_anon_rmap(struct folio *folio, struct vm_area_struct *vma) { void *anon_vma = vma->anon_vma; - struct folio *folio = page_folio(page); VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_VMA(!anon_vma, vma); From patchwork Mon Oct 2 14:29:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13406287 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 107ECE784B3 for ; Mon, 2 Oct 2023 14:30:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0FA3F6B0159; Mon, 2 Oct 2023 10:30:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 00CE66B015A; Mon, 2 Oct 2023 10:30:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D2CC06B015B; Mon, 2 Oct 2023 10:30:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C2B626B0159 for ; Mon, 2 Oct 2023 10:30:03 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 85BC2A02D5 for ; Mon, 2 Oct 2023 14:30:03 +0000 (UTC) X-FDA: 81300755886.29.7DF357F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 941231C0020 for ; Mon, 2 Oct 2023 14:30:01 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Dz583Cb0; spf=pass (imf21.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696257001; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Tnnd1i96YoBpg2Wk1+SUJX4WepSITGLgtW2LVCbh8ow=; b=Fjod7cr78Y0ZarSZJhAMceWMwOpQ7IQmyK4ntzbfNtQ8gw95RtMYJKb59gmdp+cSn3nS6F Qs2mpQzeRkAHlRFkIpiK+bqUOnE6L7hVGYtWgHV0LJnBNb5RDIur4wnx5dXo5ipnSHZyno 1z58/m08KtDwF7fSABiUxd12q3FfNcA= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Dz583Cb0; spf=pass (imf21.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696257001; a=rsa-sha256; cv=none; b=NBjbdetjGYpH7atvq1Y4jLPW/k1LIBcy3o1KpcYSFIhK4Ggdbv1eW0MRSBWcU0VGQ4MhWm ookRy28jCATrpUkOgVnSLo+t8y6hCpkRJNXAAfdQb3+jgSDSvDbDz24t3gNIwn4N1X3wD7 1jdSDtulpaTJjlnR0G3/tTBi9ML+C1Y= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1696257000; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Tnnd1i96YoBpg2Wk1+SUJX4WepSITGLgtW2LVCbh8ow=; b=Dz583Cb02a88sh28ANAiA6t8Vm29OxT3+WjHqCNOmETP5EJ0KT6RRYX7EaM6iuBoBICchN 3dQYBAIzwucEPgo+9jNRXapV/n3ii/s995fWhBAoXPEvH6sSf3SYO0z0Y/wn3BT9JkObzd oRrF9zWeKcrlbdP+OKMRpdP28o2LxRk= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-596-SD06yepTPyaOIWqDKjRAKw-1; Mon, 02 Oct 2023 10:29:56 -0400 X-MC-Unique: SD06yepTPyaOIWqDKjRAKw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 805873826D2C; Mon, 2 Oct 2023 14:29:55 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.194.19]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5F47B1005B90; Mon, 2 Oct 2023 14:29:54 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Mike Kravetz , Muchun Song , Suren Baghdasaryan Subject: [PATCH v1 3/3] memory: move exclusivity detection in do_wp_page() into wp_can_reuse_anon_folio() Date: Mon, 2 Oct 2023 16:29:49 +0200 Message-ID: <20231002142949.235104-4-david@redhat.com> In-Reply-To: <20231002142949.235104-1-david@redhat.com> References: <20231002142949.235104-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspamd-Queue-Id: 941231C0020 X-Rspam-User: X-Stat-Signature: g6ezx5r8zdomnryuhns9o8spyex1p7ia X-Rspamd-Server: rspam01 X-HE-Tag: 1696257001-168390 X-HE-Meta: U2FsdGVkX18qhxdGLP8uKSx4/m978y6K73CHLpq//ISRVFqdS7r0fj8rZrFTJe1JciWClxx9Lupj7YCCgEJKdeZNN3zAdSJBh2lzILXfJw1fMBzDnT2hb2qC3Y6nuEbDZ8UnPYKXBb9UzcG1xiumIbBI2ZBTrXkO9dqn810Q2WeKgMlieB4xhmlIfB0QWyKFT54CxdvbpzNN+NpsPya5c9bbU0woRilZmSzxSkr4zaCJI1UhPbOM/OHFcpYUHLwVL0pncHx3hymHNo34pz1KvePgyAQEx66rZSJXnjYhouo88ifALLAjPYImbLW/WJ4RQbAYI0IRwQGMs/ejQzIwwFbxZuedNyuzbvYHPIDwxVsVNqrucBO4NNgvvGbnTlyrK5n784o9hFU6TEiBPoJQVaqhTcd/Dzfj/uULrevLk6UCp98siqmUbJH9+OXR7X1kXPOGEVUeP+iJzFYH/VlJEy6SVGfYGbHRSGOMrKIz4/WNBeE3znZU81TIKjewlWCi15aQR8qGLoLiHN1GYLrVYRz7FzOCQ6D7pcrM8xXsfYevA8inK5ycyx5lnVl5LctKPFYKy7QsVFS4sSStCLEOHwAMSgdqzCPxbvyzKXAzcSvLL+WU5S+3Fff7cr6t0weUfdF3QVz+bE7bBdA0a08hk3dgTmQdlZdFvDDITwAHsX2mZrIxbsKmlr135lYdpyZAz2yxywhp6wSys2c5EkmbhbVQcxQbKQ86sYLYGQxiOLvGs/zLd0ZNRh7w50HCkgSl43q0GItZ2z3JMfYkCK79+fUtu2+Q/p8TwiC7Od2metKFP0QFI2ns/3gRSD1QYepW54BALUJX/1J7fugT+B0L7Kj2BQuCGS5+WerjoeAoWGLWoqa1Q0h5DK+ksBZa8fMiTPTjWoW5Bgt18QmgO/lQYLQpiKpl1cTLz/MXLKKhemO8icVdVunIzhc3tmLRnDBTsGEhbz0xHpk0BbUlMxj aG5yts8u aq7Z+AbByREgasnL/jeRoEblldZwolbSzX6UvTNaMCFCxhOArfq4VBdsOUY795aBfy+OyipzenOV1PXaSu6UNAwxfQaUPlpIkOS06h4cvrVa2Sir110wHInjLHXUcp3BT011vqNt/mH+Rx1dvoz5GXqX5WwIdbU+qjuUsomwlJFpf5E1k0hqqtzoVTOHt2axdc1QUngi59LDzOr9hsNlcI4wBww== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Let's clean up do_wp_page() a bit, removing two labels and making it a easier to read. wp_can_reuse_anon_folio() now only operates on the whole folio. Move the SetPageAnonExclusive() out into do_wp_page(). No need to do this under page lock -- the page table lock is sufficient. Signed-off-by: David Hildenbrand --- mm/memory.c | 88 +++++++++++++++++++++++++++-------------------------- 1 file changed, 45 insertions(+), 43 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 1f0e3317cbdd..512f6f05620e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3358,6 +3358,44 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) return ret; } +static bool wp_can_reuse_anon_folio(struct folio *folio, + struct vm_area_struct *vma) +{ + /* + * We have to verify under folio lock: these early checks are + * just an optimization to avoid locking the folio and freeing + * the swapcache if there is little hope that we can reuse. + * + * KSM doesn't necessarily raise the folio refcount. + */ + if (folio_test_ksm(folio) || folio_ref_count(folio) > 3) + return false; + if (!folio_test_lru(folio)) + /* + * We cannot easily detect+handle references from + * remote LRU caches or references to LRU folios. + */ + lru_add_drain(); + if (folio_ref_count(folio) > 1 + folio_test_swapcache(folio)) + return false; + if (!folio_trylock(folio)) + return false; + if (folio_test_swapcache(folio)) + folio_free_swap(folio); + if (folio_test_ksm(folio) || folio_ref_count(folio) != 1) { + folio_unlock(folio); + return false; + } + /* + * Ok, we've got the only folio reference from our mapping + * and the folio is locked, it's dark out, and we're wearing + * sunglasses. Hit it. + */ + folio_move_anon_rmap(folio, vma); + folio_unlock(folio); + return true; +} + /* * This routine handles present pages, when * * users try to write to a shared page (FAULT_FLAG_WRITE) @@ -3444,49 +3482,14 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) /* * Private mapping: create an exclusive anonymous page copy if reuse * is impossible. We might miss VM_WRITE for FOLL_FORCE handling. + * + * If we encounter a page that is marked exclusive, we must reuse + * the page without further checks. */ - if (folio && folio_test_anon(folio)) { - /* - * If the page is exclusive to this process we must reuse the - * page without further checks. - */ - if (PageAnonExclusive(vmf->page)) - goto reuse; - - /* - * We have to verify under folio lock: these early checks are - * just an optimization to avoid locking the folio and freeing - * the swapcache if there is little hope that we can reuse. - * - * KSM doesn't necessarily raise the folio refcount. - */ - if (folio_test_ksm(folio) || folio_ref_count(folio) > 3) - goto copy; - if (!folio_test_lru(folio)) - /* - * We cannot easily detect+handle references from - * remote LRU caches or references to LRU folios. - */ - lru_add_drain(); - if (folio_ref_count(folio) > 1 + folio_test_swapcache(folio)) - goto copy; - if (!folio_trylock(folio)) - goto copy; - if (folio_test_swapcache(folio)) - folio_free_swap(folio); - if (folio_test_ksm(folio) || folio_ref_count(folio) != 1) { - folio_unlock(folio); - goto copy; - } - /* - * Ok, we've got the only folio reference from our mapping - * and the folio is locked, it's dark out, and we're wearing - * sunglasses. Hit it. - */ - folio_move_anon_rmap(folio, vma); - SetPageAnonExclusive(vmf->page); - folio_unlock(folio); -reuse: + if (folio && folio_test_anon(folio) && + (PageAnonExclusive(vmf->page) || wp_can_reuse_anon_folio(folio, vma))) { + if (!PageAnonExclusive(vmf->page)) + SetPageAnonExclusive(vmf->page); if (unlikely(unshare)) { pte_unmap_unlock(vmf->pte, vmf->ptl); return 0; @@ -3494,7 +3497,6 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) wp_page_reuse(vmf); return 0; } -copy: /* * Ok, we need to copy. Oh, well.. */