From patchwork Wed May 8 22:40:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13659216 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C638C04FFE for ; Wed, 8 May 2024 22:42:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B23E6B009E; Wed, 8 May 2024 18:42:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 962FB6B009F; Wed, 8 May 2024 18:42:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8033F6B00A0; Wed, 8 May 2024 18:42:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 624AC6B009E for ; Wed, 8 May 2024 18:42:29 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 1922F40E64 for ; Wed, 8 May 2024 22:42:29 +0000 (UTC) X-FDA: 82096704018.24.780892F Received: from mail-pg1-f174.google.com (mail-pg1-f174.google.com [209.85.215.174]) by imf01.hostedemail.com (Postfix) with ESMTP id 4469040018 for ; Wed, 8 May 2024 22:42:27 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ibHvdWR5; spf=pass (imf01.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.215.174 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715208147; a=rsa-sha256; cv=none; b=Tp6yT1ceXclTgTuqvt/3pOaBhqwmTFQ8VsFz92Bjh0A0RUD6MFap2bBmxWimMWQbi0N6wr cXttxXLt20wn6VIeIEkC6I/bNFykFQ9dxUatOVbpEDF9qHHdgg51yiSO3VMCgtHEV8lxj+ kfwPIam+5ulls/DOztNPIjykqgbGRL8= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=ibHvdWR5; spf=pass (imf01.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.215.174 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715208147; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SEA7q2xvrQmRo9zBznhNS6ApXjfQDgq/4gBxPbR9Hqk=; b=OEBFdM+8VloBirws3g92MOu92M4A2Hn4ilJ9n21oL1MaNAiqIaXlKUPukalSO6ILB0B6Dy iCslyC9EZi3JMZA2ct6oAVeVYMkjasYW+eKq6Vd0mZg71Agp6TMp5OTIcSbGP+mpX6uMCY FEdgdQ3FEo1bpwJUc7pORlz18BaBHwQ= Received: by mail-pg1-f174.google.com with SMTP id 41be03b00d2f7-5bdbe2de25fso205177a12.3 for ; Wed, 08 May 2024 15:42:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1715208146; x=1715812946; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SEA7q2xvrQmRo9zBznhNS6ApXjfQDgq/4gBxPbR9Hqk=; b=ibHvdWR5OonSOBvkEPoHVVMlk6KqPBEqP8Fcb9lInCGxRJ6KFxBguqxdGBTVElCX0H dG0PFOENwc73pbOOn5vCzWPlZLhDdL5d1Hjhwccci1qW2SsdYR4kvNQQ3e32bNK76K6q +5tgZHf2qf1IqHxIhVFabPpcpBnGxWQ4Wsd8+oO1kYXClBp4KExAj4o7NeB/TWREJqcp rmP14hTDKKg/odINj10em5ID1yWOLj2GQzDIrPHO6IuVkzBpJ2u4CmQ+DLpuM/FWcxDW ULhS08l4onCNyysz+ClINDXrmVw2ziphV/aHhCg7Ys0/vXZ4bszPlzFz7F+Tm4gpQp/J pp7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715208146; x=1715812946; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SEA7q2xvrQmRo9zBznhNS6ApXjfQDgq/4gBxPbR9Hqk=; b=CGgyKr3dtjSVHtWVu540jK9cGQ4McqVLHIFu8ZDJ/UQqNrU+8/XOvMOz99GZMilWcS dIyyxf/IJKs6ekxdeprQZKYIvnL4VyxxxJGDJ81GDjeeYqQP61/5HGnjdB5Tgp9hkHgX kCPOPZ+m4QV2l5U3syDy2mPCFbZkANT5LHP+DiafzZQ5ipNXrsG3LQBtqIkUvFN7XoYi 2a0CDoTi2BIesOf6xbHdPu0rtC95T/247UmpivZEa/T216GJb7r9J9EQg17VKz8p3+Cy gslomtXuAEqZw38jPfvusR2HNzyonDpOjpvXB7ULllB3hIPBpKK2GgdBBFl8kv6gqORb TOdg== X-Forwarded-Encrypted: i=1; AJvYcCVlatOYooyUWeUy7jNMPQs6CfC8IjlUMWimz15c/nGggk0C4mIAcpJEDBRcJJbfoyAuPBNABcGKBA9YmbzG6oCinFQ= X-Gm-Message-State: AOJu0YxV74simNpni2T6+n0Hc5Ffyjyp2AtNM/c2nKoV1zE9oD9fCPHI /Mj2tLghCUr1ax+xyemytd2nEbFX5+6CL2AFgb6ObnsKmMffhoPV X-Google-Smtp-Source: AGHT+IGq29rmz9yqnIyAWg43yTePii4lbT2fQtdkcVHel1ug/Dy/4QuBgQDgX9YahoaNoa9CKKLOjw== X-Received: by 2002:a17:90a:8811:b0:2b3:28df:92bb with SMTP id 98e67ed59e1d1-2b6165a62eemr3793409a91.13.1715208146212; Wed, 08 May 2024 15:42:26 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-2b67126b666sm102082a91.34.2024.05.08.15.42.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 15:42:25 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v4 6/6] mm: swap: entirely map large folios found in swapcache Date: Thu, 9 May 2024 10:40:40 +1200 Message-Id: <20240508224040.190469-7-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240508224040.190469-1-21cnbao@gmail.com> References: <20240508224040.190469-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Stat-Signature: a1ag9sxaw67rudqk9i1gpq69dymgp5ju X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 4469040018 X-HE-Tag: 1715208147-463164 X-HE-Meta: U2FsdGVkX19juY2ZnubYppaZ2h2HlCMb1wFvFjA9KHcNTmTueLU1IlfKs4jrF9tn5Fc+N1AyQ691qUNZgNuYiNr6FhvZOO+AbcUN68NvcgwMEjPHB24MfCwkGzF2xvXEqoZg3MLat3y5JxeODHx/1FVQpdtzdqF07n5tmi7o77eJnY1gI4d6/fe4uIPWPcc7huXv38oFDeL8UBO6jNtpR8u64hoWrVI5Dv9EEfMLJ6TBFvkYZ1fEY5JGPbVfwPBQr0ZWrFr3EY6YjmUurH56InRL0wPwlBBN3rxHfxVua6+d9tCvSn7T6PO3j7o04p/gNZ6rXy/3vmLpk8H4D13Npo94n8SN1GsG1p8/MbIxCOKHbybACTJvf1Qm1WgSs4wtHOnUqPfM4L+NIbJN6J6VGteDND8/7XxKW700fBLy+5ofJ0JQKSbai0UrC8Z6+Jb6ra2pNnRAZmoUfaVGunBxBkCY9atHvPH93/ResFgJHSlv4a2+qnX6hGFdqUA7IPbKGirWaaTvzqhQhIehY+igWq7i0Ba5DOshA35bIaxIH5+PZYD6CqWPbZCfGmpgdAAPsbQODb3FB/ZvYznrNDYO3s/1BnHn1WAfiziuSjf8mpws751QF2JGAOX0ln3+iejIY6W77yefn338fbD+EFCbmt+nWnKQaejv4pYSghCT5S4clRuNsjQgwLjPqsvI28DOOUJtkj09Drxh8KXQTxi0Lqi5bphFThaM2U1Mhe4sdurD2+t8vu4DvOcKAbXA5DPCVrMa1eJ2xIHr1th0T9HJmMG9Lk8AyJw+tag5M8JTS9nUAqY6qNDK0DDoUNV7MUIGlu96CN/y92ijJLnBzK6j82Jg8iQNSbqlX26wrUff9HUt6vosXegADxUDlDBNr1jY+dp8Gw3FPSXlfuMYvzzpBjFcwIPmRE77KZ/QMRA1nJ6tRYHisy27M5OahfjZlqiPFH/zpHjZwNPkjDNZpxE YEot6hza gdDmIufW2DM9jieDt4XwTpUoPiQjPaXndnLAFl6PfOc0wfiiTS2HieflTWwC125GWzNHMSp8E95HHzYiJ/t50fm9IVK7gl16tADJnk3qhtb0aSyx9r70HiJnIcA9XXSqQDGHnLNtVYTK/aEhkUxP2eYRznqrqfLT+ZKSxWQTY5oTyULUIEOtCXMPeHf2+dRL4H/cv/ANBN7uCJTJ+IJILnfbE4puBgQjWcASUBpxt3PJWreZ6NHOz8MaTfbAhMqQN/xV0eygaxWpaDC38GwIQQ0iivus0FL1Dfpnybn6nhEe5jFf+SPqEUKY2NAQ9/6Aw4J6pdRniAlNhnjao/Yl/IrvPHYze8RVY7Svy2dE42BqcUcAexGWukB3jsg2D9BPHqIz59kbxoXoN7rutRLfTjKNpfhaCF/4kOI9e5i+ysAT5bHeeRspm6238uQs9TPbe+s2H7MfvSYF7QaJnRqQhBcaxDQJfce65G2c8A8JkUMS3TwjNgHO9YA+CFItTMoTZ/eLD67ac/B68NeYZ4SYtTCSg5df2UWSFAtvWWmcuIpULCl2f2n6ERQNf4xYxNwNv+N6CKbItMCkOrjA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han When a large folio is found in the swapcache, the current implementation requires calling do_swap_page() nr_pages times, resulting in nr_pages page faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Reviewed-by: Ryan Roberts Reviewed-by: "Huang, Ying" --- mm/memory.c | 59 +++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 48 insertions(+), 11 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index d9434df24d62..8b9e4cab93ed 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3968,6 +3968,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) pte_t pte; vm_fault_t ret = 0; void *shadow = NULL; + int nr_pages; + unsigned long page_idx; + unsigned long address; + pte_t *ptep; if (!pte_unmap_same(vmf)) goto out; @@ -4166,6 +4170,38 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } + nr_pages = 1; + page_idx = 0; + address = vmf->address; + ptep = vmf->pte; + if (folio_test_large(folio) && folio_test_swapcache(folio)) { + int nr = folio_nr_pages(folio); + unsigned long idx = folio_page_idx(folio, page); + unsigned long folio_start = address - idx * PAGE_SIZE; + unsigned long folio_end = folio_start + nr * PAGE_SIZE; + pte_t *folio_ptep; + pte_t folio_pte; + + if (unlikely(folio_start < max(address & PMD_MASK, vma->vm_start))) + goto check_folio; + if (unlikely(folio_end > pmd_addr_end(address, vma->vm_end))) + goto check_folio; + + folio_ptep = vmf->pte - idx; + folio_pte = ptep_get(folio_ptep); + if (!pte_same(folio_pte, pte_move_swp_offset(vmf->orig_pte, -idx)) || + swap_pte_batch(folio_ptep, nr, folio_pte) != nr) + goto check_folio; + + page_idx = idx; + address = folio_start; + ptep = folio_ptep; + nr_pages = nr; + entry = folio->swap; + page = &folio->page; + } + +check_folio: /* * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte * must never point at an anonymous page in the swapcache that is @@ -4225,12 +4261,12 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * We're already holding a reference on the page but haven't mapped it * yet. */ - swap_free(entry); + swap_free_nr(entry, nr_pages); if (should_try_to_free_swap(folio, vma, vmf->flags)) folio_free_swap(folio); - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - dec_mm_counter(vma->vm_mm, MM_SWAPENTS); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); + add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages); pte = mk_pte(page, vma->vm_page_prot); /* @@ -4247,27 +4283,28 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } rmap_flags |= RMAP_EXCLUSIVE; } - flush_icache_page(vma, page); + folio_ref_add(folio, nr_pages - 1); + flush_icache_pages(vma, page, nr_pages); if (pte_swp_soft_dirty(vmf->orig_pte)) pte = pte_mksoft_dirty(pte); if (pte_swp_uffd_wp(vmf->orig_pte)) pte = pte_mkuffd_wp(pte); - vmf->orig_pte = pte; + vmf->orig_pte = pte_advance_pfn(pte, page_idx); /* ksm created a completely new copy */ if (unlikely(folio != swapcache && swapcache)) { - folio_add_new_anon_rmap(folio, vma, vmf->address); + folio_add_new_anon_rmap(folio, vma, address); folio_add_lru_vma(folio, vma); } else { - folio_add_anon_rmap_pte(folio, page, vma, vmf->address, + folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, address, rmap_flags); } VM_BUG_ON(!folio_test_anon(folio) || (pte_write(pte) && !PageAnonExclusive(page))); - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); - arch_do_swap_page_nr(vma->vm_mm, vma, vmf->address, - pte, vmf->orig_pte, 1); + set_ptes(vma->vm_mm, address, ptep, pte, nr_pages); + arch_do_swap_page_nr(vma->vm_mm, vma, address, + pte, pte, nr_pages); folio_unlock(folio); if (folio != swapcache && swapcache) { @@ -4291,7 +4328,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } /* No need to invalidate - it was non-present before */ - update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); + update_mmu_cache_range(vmf, vma, address, ptep, nr_pages); unlock: if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl);