From patchwork Thu Jan 25 19:32:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13531559 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF8C7C47422 for ; Thu, 25 Jan 2024 19:33:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5282A6B0081; Thu, 25 Jan 2024 14:33:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D82D6B0082; Thu, 25 Jan 2024 14:33:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 378B16B0087; Thu, 25 Jan 2024 14:33:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 23D786B0081 for ; Thu, 25 Jan 2024 14:33:49 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id ECC001C18CC for ; Thu, 25 Jan 2024 19:33:48 +0000 (UTC) X-FDA: 81718833336.23.B7E9FD7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf19.hostedemail.com (Postfix) with ESMTP id 4DA121A0013 for ; Thu, 25 Jan 2024 19:33:47 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cmwXi7+u; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf19.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706211227; a=rsa-sha256; cv=none; b=Zolls5wncnOA4hI1EcdlNr0J7uBcPY4cS4o12SqGVuOcJARoy8QWle5mru0n8ivPfTKvM2 VtgN8bvajXiEPoAV3a/v1SxIikATjY9Eo2yFYCyPknHZmI/vURXm4ayKQyFVHEWrHciwD1 jkVMEOtMJZRlofPaUTvPG9ARjc4be4c= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=cmwXi7+u; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf19.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706211227; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3s1Jv/6P5yjhblLF2udczJAUFVvnCKFeQLjRuwj24+c=; b=GIXRBViPVmk8m9maYglhqK6q/lD/nXwmvnKuA0CaXVRASETrpfMSaWPi5bkC0uKIraJOK4 JlJZbUlE1Ulw1QUrMA43Ag6TZJVqG+26Zg1jshSlhkNOKVP1wRV7h84EFr9N4iZJCai1v/ dt12igxYLgZ8PDVDcRwkWn+8e9KtgJo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706211226; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3s1Jv/6P5yjhblLF2udczJAUFVvnCKFeQLjRuwj24+c=; b=cmwXi7+ucx0JaGuGm1FvBL5hHGaPU1IOUxt0b2w8YWWRcf2T9Yk1CsDtR5nph4dhqYaUgR MmcoAZvZz9EDORTYWv5tp5mtOocpGUKhv8OaSB1NnjFRtTabts+srYjyMtydbO5Tfmkm2Z w+3FtRMD3CoQhkPlALAdqgx09vOwN8s= Received: from mimecast-mx02.redhat.com (mimecast-mx02.redhat.com [66.187.233.88]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-499-493L1nvTPrqpYIPc57u-fA-1; Thu, 25 Jan 2024 14:33:42 -0500 X-MC-Unique: 493L1nvTPrqpYIPc57u-fA-1 Received: from smtp.corp.redhat.com (int-mx09.intmail.prod.int.rdu2.redhat.com [10.11.54.9]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 34A868432A1; Thu, 25 Jan 2024 19:33:41 +0000 (UTC) Received: from t14s.redhat.com (unknown [10.39.193.154]) by smtp.corp.redhat.com (Postfix) with ESMTP id 074EA492BC9; Thu, 25 Jan 2024 19:33:34 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Russell King , Catalin Marinas , Will Deacon , Dinh Nguyen , Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , "David S. Miller" , linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org Subject: [PATCH v2 11/15] mm/memory: factor out copying the actual PTE in copy_present_pte() Date: Thu, 25 Jan 2024 20:32:23 +0100 Message-ID: <20240125193227.444072-12-david@redhat.com> In-Reply-To: <20240125193227.444072-1-david@redhat.com> References: <20240125193227.444072-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.9 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 4DA121A0013 X-Stat-Signature: hncaajgmt84mesfgdahs894bfxhj4byd X-HE-Tag: 1706211227-601073 X-HE-Meta: U2FsdGVkX1+11Vf8UL519v4j91HdPGYn15A8REN6fcvt6SguCxrqlc8g1h48dvgizTsZrHvAcuHQlpW2kjvLRShlvjYI7L8Ewl2MRAlqiUS/oPH679JlBt+PswEOSFSXmqAQdhz33CslJDCeJdKqIZ7+wfP6Yf6p0XJmiT5bAlZn7StxRVn3Tzo891GWINoo6RRLDvrKehJDVNADxDdfH9STHhWpQjD1qMuL/uvcQWQqb+HrKqTf6k+Tvh4CR9vjmu4FWF2sreyg5Scl3LCBOsAbDnBr4NRnyiKExhyMW2HVyWGR51q/WgVljuBIMqTjfYDAHTo+BgtZtLtYVWMJQQjMjs8l9N5vl2gEHZTSkIvgA2+05QcdMplaFSX8wXfucfF1wgL9vEsgWtb70F4iu6xiUvkNgHNZx5miFelNpPsU+7QCAM603plvLBX6Wg2fZm6fBpUWzuv0i6HorWWgK3iNXfSBgeQaCMcnzh3IShl3CwHIEohrIvqPSzOPU4x8/MFR1kEF4IedP6wXkPUpCGlefYVFja2Fq5MRJfdydtYuxiTyQoMuDR9knjsCnHEMD9GXD6T2LIAfdFN0MmqYkskk07+2dOErk5zw88SoGoV3abrf2y+TSFrCZKDr+HlRm06cRsSGriwdMBc2WERR2kzz2wee4UAJei35/Gpa11iwqxnWpTolzAlXXc3y8EqCMyTk7+oFJzbHR9MMCN7maPQA6hQF3Z2qi8JuS5toNtkas/0PrYyd0Szi/zxYk6Nt2zruVRThe0rNIj42cenT8VVV2+Hm0jpy+iizOfwJWFu8ebj2pHaGrKLrEaWEsP/bAKTPi7rSFwWfxefQyhr5F/YKb+hScTD46iSmTbW1ZbKLvV6jJPs4iR8lGDhwf1404KQuVGLRrssBiN/MeFGigPPcgY36SW6x6QWkn1UVHqa6iehOnjPmY3XnneqE6NaYvEB0gq4vekmrg7TjZSB 2TJx0Ak5 57mS0yx56A2X8fCYU9ZGDzNAq3+RYxltijOUkfus4lfUv6i9uct3LrszLy/KuOa2o7mzYxX204uadjgpHcNJJmGIzP2MjRB352vHUiG8bkoX+/gz8ItlbglaFXg1IFhiOJZZKLbLvYcXbZJ/rU/oOYo3l2NXOVLwDEP5b4XE/KbqcFhG2Y1mRRkrSuIeJEurysmoJJ3kZmyIMZj7sI0vBEIIorNBie8H6zX7KTiAr7IzzPAmFQ9kpPIPuTZbsUWRtKW0zgdERYQCoopYF2Ou6akGJ/Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's prepare for further changes. Reviewed-by: Ryan Roberts Signed-off-by: David Hildenbrand --- mm/memory.c | 63 ++++++++++++++++++++++++++++------------------------- 1 file changed, 33 insertions(+), 30 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 7e1f4849463aa..10fc14ff8e49b 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -930,6 +930,29 @@ copy_present_page(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma return 0; } +static inline void __copy_present_pte(struct vm_area_struct *dst_vma, + struct vm_area_struct *src_vma, pte_t *dst_pte, pte_t *src_pte, + pte_t pte, unsigned long addr) +{ + struct mm_struct *src_mm = src_vma->vm_mm; + + /* If it's a COW mapping, write protect it both processes. */ + if (is_cow_mapping(src_vma->vm_flags) && pte_write(pte)) { + ptep_set_wrprotect(src_mm, addr, src_pte); + pte = pte_wrprotect(pte); + } + + /* If it's a shared mapping, mark it clean in the child. */ + if (src_vma->vm_flags & VM_SHARED) + pte = pte_mkclean(pte); + pte = pte_mkold(pte); + + if (!userfaultfd_wp(dst_vma)) + pte = pte_clear_uffd_wp(pte); + + set_pte_at(dst_vma->vm_mm, addr, dst_pte, pte); +} + /* * Copy one pte. Returns 0 if succeeded, or -EAGAIN if one preallocated page * is required to copy this pte. @@ -939,23 +962,23 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, pte_t *dst_pte, pte_t *src_pte, unsigned long addr, int *rss, struct folio **prealloc) { - struct mm_struct *src_mm = src_vma->vm_mm; - unsigned long vm_flags = src_vma->vm_flags; pte_t pte = ptep_get(src_pte); struct page *page; struct folio *folio; page = vm_normal_page(src_vma, addr, pte); - if (page) - folio = page_folio(page); - if (page && folio_test_anon(folio)) { + if (unlikely(!page)) + goto copy_pte; + + folio = page_folio(page); + folio_get(folio); + if (folio_test_anon(folio)) { /* * If this page may have been pinned by the parent process, * copy the page immediately for the child so that we'll always * guarantee the pinned page won't be randomly replaced in the * future. */ - folio_get(folio); if (unlikely(folio_try_dup_anon_rmap_pte(folio, page, src_vma))) { /* Page may be pinned, we have to copy. */ folio_put(folio); @@ -963,34 +986,14 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, addr, rss, prealloc, page); } rss[MM_ANONPAGES]++; - } else if (page) { - folio_get(folio); + VM_WARN_ON_FOLIO(PageAnonExclusive(page), folio); + } else { folio_dup_file_rmap_pte(folio, page); rss[mm_counter_file(page)]++; } - /* - * If it's a COW mapping, write protect it both - * in the parent and the child - */ - if (is_cow_mapping(vm_flags) && pte_write(pte)) { - ptep_set_wrprotect(src_mm, addr, src_pte); - pte = pte_wrprotect(pte); - } - VM_BUG_ON(page && folio_test_anon(folio) && PageAnonExclusive(page)); - - /* - * If it's a shared mapping, mark it clean in - * the child - */ - if (vm_flags & VM_SHARED) - pte = pte_mkclean(pte); - pte = pte_mkold(pte); - - if (!userfaultfd_wp(dst_vma)) - pte = pte_clear_uffd_wp(pte); - - set_pte_at(dst_vma->vm_mm, addr, dst_pte, pte); +copy_pte: + __copy_present_pte(dst_vma, src_vma, dst_pte, src_pte, pte, addr); return 0; }