From patchwork Wed May 29 08:28:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13678367 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4350C25B7E for ; Wed, 29 May 2024 08:29:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 59D596B00AA; Wed, 29 May 2024 04:29:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 527666B00AB; Wed, 29 May 2024 04:29:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C6EA6B00AC; Wed, 29 May 2024 04:29:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1A0B76B00AA for ; Wed, 29 May 2024 04:29:32 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id BC550C0971 for ; Wed, 29 May 2024 08:29:31 +0000 (UTC) X-FDA: 82170759342.11.90E2B98 Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by imf26.hostedemail.com (Postfix) with ESMTP id E9DB5140008 for ; Wed, 29 May 2024 08:29:29 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=fcImPoCn; spf=pass (imf26.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1716971370; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kxE2mAKSIadz3iJIa+4kx5y9ZNqImGpGMb9QcvFrG9A=; b=tlUc9+WerhpSFvP1M+LO8rmPYiNPSbtwE7YEaVvzVLm0B2QG/Xo8jyfsiFeThSp9pxnsBC zCxyyFjPe8f9gS3BYvYCAfZfTwpyYpj8ReVnWwEJCuHovr4RxgqWyfBzYJ8PSx+q0Lt0Bp pSxZDeuwwXjfhLV4NI/pJJVuLTHMaUI= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=fcImPoCn; spf=pass (imf26.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1716971370; a=rsa-sha256; cv=none; b=VlfsGPCPDlo2ZrVXmPSIUJqTkv5MjVy0cT7ECuMEAF35pdWAqPrPmqhNeryiO7rUjCjt9K aGiWTa8ncW4+a5z9RppCR+AdtiaPRdGJt5O9+C7OMSPZCMRt4AiHCLx3VJJSiJOa5ISMvX yaJRb7za5HzDxuYwxNY5UvfOqhhUlbk= Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-6f4ed9dc7beso482539b3a.1 for ; Wed, 29 May 2024 01:29:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716971369; x=1717576169; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kxE2mAKSIadz3iJIa+4kx5y9ZNqImGpGMb9QcvFrG9A=; b=fcImPoCnybjRJ1JovokCu6rmF8kAbylDUZkWnodSIGF2mNE7JVHJReYeiUQlKxo6xg Apz58z4bB0C7jICaFyi5PWR7Go8aZH+edeek1tiXDrRu+hrubZIlAruP/opGZI6cH7t4 VTnCwXSuC8CVGpCNH60yfgi4N8K4O8lIf5oq9E1c+JTgJA87J2mmGo2fYCONA0YzWuqI 7tS5vNsCktAoKELVhT9b3H3KL0fflVvd/RdaJqPNQJG2myyh1RmU+aGLpkOSfoCANSSA PBCLGmJ5AxGNmvrgJnV3wmsHtnqJNKw9pslZWBqwcnqnvMrSm/pB9N3iK7Vf5vk/GTfs S16w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716971369; x=1717576169; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kxE2mAKSIadz3iJIa+4kx5y9ZNqImGpGMb9QcvFrG9A=; b=q9z65IODpgykV4yfjAEC62MTExlCQfNtNfbpthRObclPGf5clfvOlmuaielISyQvw9 jrb5cNFFwkjtx9Aaa3nkp2M/LljtXLqiNuzPmXbCrXnvKUnuXtIwzgvQq+RYmO3+K33p 1PTDL6+OHFdkSwUl11KD1Iwsu9IgF1Aj+zEDKZ/umKfYd7W1rQ65UCPA48e5tRCb1OJh Yp9ZKlHix6mkNquKVHq4wH5qP0WsNIuO73IcIO08SZIl3zha3dt8mHLjmnbHFClQVN7y /e0pqHZ5rNwrOeEvSb5nuCYpA9UftQrJPIU1z0GQAVZEu22VMcWHTPBPmRNO46xWfNrU UNJA== X-Forwarded-Encrypted: i=1; AJvYcCV7D/166oPE3BAOdfYf1IuXmQ5zEPJA6A6CGAv+gxRjC4QNdp+s2IHU7PyAY6MEwJLk6EEUcUlWgQs9i6KVtjVS3iA= X-Gm-Message-State: AOJu0YzQnbGMAGzO3JO09oPAu5WXTgKp91VnyE3wzQ1u4EyWYXfUFGHQ qYRhJHWpcb2QpvQHqaRz0/1bDmeRqk89UxwJHUPmgG7kIo9fUSWz X-Google-Smtp-Source: AGHT+IEbSGge22ETQ01VYggngJlR3UpTZl4VLPzOebNpH6caOUJ2L4BysvKHtwvqOmVKwJG58Oyn5g== X-Received: by 2002:a05:6a00:2fd1:b0:702:1d51:98f7 with SMTP id d2e1a72fcca58-7021d519c44mr490666b3a.2.1716971368774; Wed, 29 May 2024 01:29:28 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-702220d948bsm174916b3a.131.2024.05.29.01.29.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 May 2024 01:29:28 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v5 6/6] mm: swap: entirely map large folios found in swapcache Date: Wed, 29 May 2024 20:28:24 +1200 Message-Id: <20240529082824.150954-7-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240529082824.150954-1-21cnbao@gmail.com> References: <20240529082824.150954-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Stat-Signature: ji6dj6ityiwgbm6r945z46n4c6955ndm X-Rspamd-Queue-Id: E9DB5140008 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1716971369-374697 X-HE-Meta: U2FsdGVkX18uz2vde7Ff6AFWZrNO+6d1JOtX2CXD7R4H4QNuBLovPHfxNFvQYBRASCoCkKcpBSUeHy7YYW8bszJhUnWRgdzi4yCIG2uMlwqIEp3+BR2DmXH4F4uwgMIy/RQ2iVje7balOYFmEdDH9eU4Zi0DwAMdobSTiJQri+lkS3ARCjtEGkD50o78LEEXsyNmStvFCDKc9FiLSkWNCNoGkx3hvHGWjuk0pjIbETaLewkCF0ASq8K3AXLLNdLLWHTdQ/y/cwK0PMoWRdZHfyPqXgSZDuwGmm2yF0aABvr8QyhdAQvekHHB1685G11lfPy2Zg5Ti74y4kDzjUWokcFN36UVzsTiAudonQP6tg3NZy2D8v5bOotGm5IlO3WmdVnXcObmK1XJLuQ7lB17JQuAdKLVjVOE3DMKzxjw1rYNV3fVwZEGnlyNA+fBy87HTvlnTVLV2LdsU9gOIQA/0I8yqGIGr/3cIhcUAZbxU33xC+TrmqjHJ3KLHgQB7rMkb5z+D7pvKuwYvYvlQDDDJ52hfkddXNj34fab1FR9D+DGfkRh9RvPjAv8dxO8NPWKlnaXQhnNa0YQSFnnnU536/fY2sBxIuU2z3belIMPZzB3/JclA+8qDPbOQQVK/ic9mOoeqW1CLASR4n/lfTtthqJKQjJXmEdfk56uVY/eb492BF/FU7/FOkAYRGCGn6qNOmAs2RKaCjItcB6ogtG+AbUzPidzkvgZaCq1CGLkc5bZOOpVlTkh5n/xT4rlXuoB4Yx7u/fOiWmJf8XnFpvZ5WXgQGwsQAwctNO/YTWbLZvrgWXzy5/QQXXCt8+11/sJPHgFOFfHRwgPhNFPJs8x1SkzkzTnGQ793Clvp4+FVUFhdkJj1Q9PyDdNy8o7cIL6dYX2DS3WMIRMA48PJ2oLTsxYJnimje+hhLqzkHTUXRF5XHvA3Uz7Cr1/UTTaQ/fYcywBABkUhVj8h+gs4La 8AWYT3y3 NgammXo/PffFhZw/juM2j7bDGKO0fjarfDOc33+Rje8mptCviINWLm/u3+s/DT9cN2gRA2hbZXD+5xYv8KRO2PLC5p9hoilpyAX0A1IDhwCPBxf2PTkV+Jh7IidfVgLWRXT1ioM2aeoI2inexmfZfx3kUuL8T7JpkWDiWPCerWiZaC5tZ4IDVMgQpA5dAYxtXLT/A7RCS/x3+WMyiUHCvqoZpaC54++pG9BwjfAinicnNVFe+vXYTtguari2j1D0M1aLp3LZSneSVezevgEJ4Bb4YwD/M8i8w3OTBF3j4TLST5c7EfmKTuHDwKFGK8qZwcG+d31pCo3W+L4lNesChg6YKwYbbO15ngWbfReLqoqES6qFVf1hszXoz39CVRBJW9g9iCQI9VmPU0v0IAJofzQ5uIICuMeYS4PtvHnwFhxjRNCBozUBku8re+rCRvdsXpn2AlcK/e5ja2gTHqulm3a0vz8ANi8O6nLZWV4rGaCCh7sKv4p1TC0iKJYh2p9gyuvo3nkp5oG8l9woKxbyLeu8pgEn6Gq8JgSHOPizpmux6h8hWZOaGNhcYvBhGSxzKVXz9hyc2H1SkOFDgmFdOt/G86gJIeEzkyC/n X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han When a large folio is found in the swapcache, the current implementation requires calling do_swap_page() nr_pages times, resulting in nr_pages page faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Reviewed-by: Ryan Roberts Reviewed-by: "Huang, Ying" --- mm/memory.c | 59 +++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 48 insertions(+), 11 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 10719e5afecb..eef4e482c0c2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4016,6 +4016,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) pte_t pte; vm_fault_t ret = 0; void *shadow = NULL; + int nr_pages; + unsigned long page_idx; + unsigned long address; + pte_t *ptep; if (!pte_unmap_same(vmf)) goto out; @@ -4214,6 +4218,38 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } + nr_pages = 1; + page_idx = 0; + address = vmf->address; + ptep = vmf->pte; + if (folio_test_large(folio) && folio_test_swapcache(folio)) { + int nr = folio_nr_pages(folio); + unsigned long idx = folio_page_idx(folio, page); + unsigned long folio_start = address - idx * PAGE_SIZE; + unsigned long folio_end = folio_start + nr * PAGE_SIZE; + pte_t *folio_ptep; + pte_t folio_pte; + + if (unlikely(folio_start < max(address & PMD_MASK, vma->vm_start))) + goto check_folio; + if (unlikely(folio_end > pmd_addr_end(address, vma->vm_end))) + goto check_folio; + + folio_ptep = vmf->pte - idx; + folio_pte = ptep_get(folio_ptep); + if (!pte_same(folio_pte, pte_move_swp_offset(vmf->orig_pte, -idx)) || + swap_pte_batch(folio_ptep, nr, folio_pte) != nr) + goto check_folio; + + page_idx = idx; + address = folio_start; + ptep = folio_ptep; + nr_pages = nr; + entry = folio->swap; + page = &folio->page; + } + +check_folio: /* * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte * must never point at an anonymous page in the swapcache that is @@ -4273,12 +4309,12 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * We're already holding a reference on the page but haven't mapped it * yet. */ - swap_free(entry); + swap_free_nr(entry, nr_pages); if (should_try_to_free_swap(folio, vma, vmf->flags)) folio_free_swap(folio); - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - dec_mm_counter(vma->vm_mm, MM_SWAPENTS); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); + add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages); pte = mk_pte(page, vma->vm_page_prot); /* @@ -4295,27 +4331,28 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } rmap_flags |= RMAP_EXCLUSIVE; } - flush_icache_page(vma, page); + folio_ref_add(folio, nr_pages - 1); + flush_icache_pages(vma, page, nr_pages); if (pte_swp_soft_dirty(vmf->orig_pte)) pte = pte_mksoft_dirty(pte); if (pte_swp_uffd_wp(vmf->orig_pte)) pte = pte_mkuffd_wp(pte); - vmf->orig_pte = pte; + vmf->orig_pte = pte_advance_pfn(pte, page_idx); /* ksm created a completely new copy */ if (unlikely(folio != swapcache && swapcache)) { - folio_add_new_anon_rmap(folio, vma, vmf->address); + folio_add_new_anon_rmap(folio, vma, address); folio_add_lru_vma(folio, vma); } else { - folio_add_anon_rmap_pte(folio, page, vma, vmf->address, + folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, address, rmap_flags); } VM_BUG_ON(!folio_test_anon(folio) || (pte_write(pte) && !PageAnonExclusive(page))); - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); - arch_do_swap_page_nr(vma->vm_mm, vma, vmf->address, - pte, vmf->orig_pte, 1); + set_ptes(vma->vm_mm, address, ptep, pte, nr_pages); + arch_do_swap_page_nr(vma->vm_mm, vma, address, + pte, pte, nr_pages); folio_unlock(folio); if (folio != swapcache && swapcache) { @@ -4339,7 +4376,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } /* No need to invalidate - it was non-present before */ - update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); + update_mmu_cache_range(vmf, vma, address, ptep, nr_pages); unlock: if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl);