From patchwork Mon Oct 2 14:29:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13406287 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 107ECE784B3 for ; Mon, 2 Oct 2023 14:30:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0FA3F6B0159; Mon, 2 Oct 2023 10:30:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 00CE66B015A; Mon, 2 Oct 2023 10:30:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D2CC06B015B; Mon, 2 Oct 2023 10:30:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C2B626B0159 for ; Mon, 2 Oct 2023 10:30:03 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 85BC2A02D5 for ; Mon, 2 Oct 2023 14:30:03 +0000 (UTC) X-FDA: 81300755886.29.7DF357F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 941231C0020 for ; Mon, 2 Oct 2023 14:30:01 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Dz583Cb0; spf=pass (imf21.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696257001; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Tnnd1i96YoBpg2Wk1+SUJX4WepSITGLgtW2LVCbh8ow=; b=Fjod7cr78Y0ZarSZJhAMceWMwOpQ7IQmyK4ntzbfNtQ8gw95RtMYJKb59gmdp+cSn3nS6F Qs2mpQzeRkAHlRFkIpiK+bqUOnE6L7hVGYtWgHV0LJnBNb5RDIur4wnx5dXo5ipnSHZyno 1z58/m08KtDwF7fSABiUxd12q3FfNcA= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Dz583Cb0; spf=pass (imf21.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696257001; a=rsa-sha256; cv=none; b=NBjbdetjGYpH7atvq1Y4jLPW/k1LIBcy3o1KpcYSFIhK4Ggdbv1eW0MRSBWcU0VGQ4MhWm ookRy28jCATrpUkOgVnSLo+t8y6hCpkRJNXAAfdQb3+jgSDSvDbDz24t3gNIwn4N1X3wD7 1jdSDtulpaTJjlnR0G3/tTBi9ML+C1Y= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1696257000; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Tnnd1i96YoBpg2Wk1+SUJX4WepSITGLgtW2LVCbh8ow=; b=Dz583Cb02a88sh28ANAiA6t8Vm29OxT3+WjHqCNOmETP5EJ0KT6RRYX7EaM6iuBoBICchN 3dQYBAIzwucEPgo+9jNRXapV/n3ii/s995fWhBAoXPEvH6sSf3SYO0z0Y/wn3BT9JkObzd oRrF9zWeKcrlbdP+OKMRpdP28o2LxRk= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-596-SD06yepTPyaOIWqDKjRAKw-1; Mon, 02 Oct 2023 10:29:56 -0400 X-MC-Unique: SD06yepTPyaOIWqDKjRAKw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 805873826D2C; Mon, 2 Oct 2023 14:29:55 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.194.19]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5F47B1005B90; Mon, 2 Oct 2023 14:29:54 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Mike Kravetz , Muchun Song , Suren Baghdasaryan Subject: [PATCH v1 3/3] memory: move exclusivity detection in do_wp_page() into wp_can_reuse_anon_folio() Date: Mon, 2 Oct 2023 16:29:49 +0200 Message-ID: <20231002142949.235104-4-david@redhat.com> In-Reply-To: <20231002142949.235104-1-david@redhat.com> References: <20231002142949.235104-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.1 on 10.11.54.3 X-Rspamd-Queue-Id: 941231C0020 X-Rspam-User: X-Stat-Signature: g6ezx5r8zdomnryuhns9o8spyex1p7ia X-Rspamd-Server: rspam01 X-HE-Tag: 1696257001-168390 X-HE-Meta: U2FsdGVkX18qhxdGLP8uKSx4/m978y6K73CHLpq//ISRVFqdS7r0fj8rZrFTJe1JciWClxx9Lupj7YCCgEJKdeZNN3zAdSJBh2lzILXfJw1fMBzDnT2hb2qC3Y6nuEbDZ8UnPYKXBb9UzcG1xiumIbBI2ZBTrXkO9dqn810Q2WeKgMlieB4xhmlIfB0QWyKFT54CxdvbpzNN+NpsPya5c9bbU0woRilZmSzxSkr4zaCJI1UhPbOM/OHFcpYUHLwVL0pncHx3hymHNo34pz1KvePgyAQEx66rZSJXnjYhouo88ifALLAjPYImbLW/WJ4RQbAYI0IRwQGMs/ejQzIwwFbxZuedNyuzbvYHPIDwxVsVNqrucBO4NNgvvGbnTlyrK5n784o9hFU6TEiBPoJQVaqhTcd/Dzfj/uULrevLk6UCp98siqmUbJH9+OXR7X1kXPOGEVUeP+iJzFYH/VlJEy6SVGfYGbHRSGOMrKIz4/WNBeE3znZU81TIKjewlWCi15aQR8qGLoLiHN1GYLrVYRz7FzOCQ6D7pcrM8xXsfYevA8inK5ycyx5lnVl5LctKPFYKy7QsVFS4sSStCLEOHwAMSgdqzCPxbvyzKXAzcSvLL+WU5S+3Fff7cr6t0weUfdF3QVz+bE7bBdA0a08hk3dgTmQdlZdFvDDITwAHsX2mZrIxbsKmlr135lYdpyZAz2yxywhp6wSys2c5EkmbhbVQcxQbKQ86sYLYGQxiOLvGs/zLd0ZNRh7w50HCkgSl43q0GItZ2z3JMfYkCK79+fUtu2+Q/p8TwiC7Od2metKFP0QFI2ns/3gRSD1QYepW54BALUJX/1J7fugT+B0L7Kj2BQuCGS5+WerjoeAoWGLWoqa1Q0h5DK+ksBZa8fMiTPTjWoW5Bgt18QmgO/lQYLQpiKpl1cTLz/MXLKKhemO8icVdVunIzhc3tmLRnDBTsGEhbz0xHpk0BbUlMxj aG5yts8u aq7Z+AbByREgasnL/jeRoEblldZwolbSzX6UvTNaMCFCxhOArfq4VBdsOUY795aBfy+OyipzenOV1PXaSu6UNAwxfQaUPlpIkOS06h4cvrVa2Sir110wHInjLHXUcp3BT011vqNt/mH+Rx1dvoz5GXqX5WwIdbU+qjuUsomwlJFpf5E1k0hqqtzoVTOHt2axdc1QUngi59LDzOr9hsNlcI4wBww== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Let's clean up do_wp_page() a bit, removing two labels and making it a easier to read. wp_can_reuse_anon_folio() now only operates on the whole folio. Move the SetPageAnonExclusive() out into do_wp_page(). No need to do this under page lock -- the page table lock is sufficient. Signed-off-by: David Hildenbrand --- mm/memory.c | 88 +++++++++++++++++++++++++++-------------------------- 1 file changed, 45 insertions(+), 43 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 1f0e3317cbdd..512f6f05620e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3358,6 +3358,44 @@ static vm_fault_t wp_page_shared(struct vm_fault *vmf, struct folio *folio) return ret; } +static bool wp_can_reuse_anon_folio(struct folio *folio, + struct vm_area_struct *vma) +{ + /* + * We have to verify under folio lock: these early checks are + * just an optimization to avoid locking the folio and freeing + * the swapcache if there is little hope that we can reuse. + * + * KSM doesn't necessarily raise the folio refcount. + */ + if (folio_test_ksm(folio) || folio_ref_count(folio) > 3) + return false; + if (!folio_test_lru(folio)) + /* + * We cannot easily detect+handle references from + * remote LRU caches or references to LRU folios. + */ + lru_add_drain(); + if (folio_ref_count(folio) > 1 + folio_test_swapcache(folio)) + return false; + if (!folio_trylock(folio)) + return false; + if (folio_test_swapcache(folio)) + folio_free_swap(folio); + if (folio_test_ksm(folio) || folio_ref_count(folio) != 1) { + folio_unlock(folio); + return false; + } + /* + * Ok, we've got the only folio reference from our mapping + * and the folio is locked, it's dark out, and we're wearing + * sunglasses. Hit it. + */ + folio_move_anon_rmap(folio, vma); + folio_unlock(folio); + return true; +} + /* * This routine handles present pages, when * * users try to write to a shared page (FAULT_FLAG_WRITE) @@ -3444,49 +3482,14 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) /* * Private mapping: create an exclusive anonymous page copy if reuse * is impossible. We might miss VM_WRITE for FOLL_FORCE handling. + * + * If we encounter a page that is marked exclusive, we must reuse + * the page without further checks. */ - if (folio && folio_test_anon(folio)) { - /* - * If the page is exclusive to this process we must reuse the - * page without further checks. - */ - if (PageAnonExclusive(vmf->page)) - goto reuse; - - /* - * We have to verify under folio lock: these early checks are - * just an optimization to avoid locking the folio and freeing - * the swapcache if there is little hope that we can reuse. - * - * KSM doesn't necessarily raise the folio refcount. - */ - if (folio_test_ksm(folio) || folio_ref_count(folio) > 3) - goto copy; - if (!folio_test_lru(folio)) - /* - * We cannot easily detect+handle references from - * remote LRU caches or references to LRU folios. - */ - lru_add_drain(); - if (folio_ref_count(folio) > 1 + folio_test_swapcache(folio)) - goto copy; - if (!folio_trylock(folio)) - goto copy; - if (folio_test_swapcache(folio)) - folio_free_swap(folio); - if (folio_test_ksm(folio) || folio_ref_count(folio) != 1) { - folio_unlock(folio); - goto copy; - } - /* - * Ok, we've got the only folio reference from our mapping - * and the folio is locked, it's dark out, and we're wearing - * sunglasses. Hit it. - */ - folio_move_anon_rmap(folio, vma); - SetPageAnonExclusive(vmf->page); - folio_unlock(folio); -reuse: + if (folio && folio_test_anon(folio) && + (PageAnonExclusive(vmf->page) || wp_can_reuse_anon_folio(folio, vma))) { + if (!PageAnonExclusive(vmf->page)) + SetPageAnonExclusive(vmf->page); if (unlikely(unshare)) { pte_unmap_unlock(vmf->pte, vmf->ptl); return 0; @@ -3494,7 +3497,6 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) wp_page_reuse(vmf); return 0; } -copy: /* * Ok, we need to copy. Oh, well.. */