From patchwork Tue Apr 2 07:32:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13613509 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4D8CC6FD1F for ; Tue, 2 Apr 2024 07:33:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7C6A66B009A; Tue, 2 Apr 2024 03:33:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 777476B009B; Tue, 2 Apr 2024 03:33:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 617A06B009C; Tue, 2 Apr 2024 03:33:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4136F6B009A for ; Tue, 2 Apr 2024 03:33:26 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 09477A059D for ; Tue, 2 Apr 2024 07:33:26 +0000 (UTC) X-FDA: 81963776412.30.9363342 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf21.hostedemail.com (Postfix) with ESMTP id 2F6171C000C for ; Tue, 2 Apr 2024 07:33:23 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=HDiojobp; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf21.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712043204; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YC/Kgduuu0t6Vvbwsn8PXX7BTHypb3qdZsKV5RTpqdo=; b=XYMXCZD2eqgHjGrYrDCSS/JGLcXYDgrfuyiHsM42mjs3I9HBN+aL9Ni6JSh3OQyvD+uQwX 66xspGYLLlqrXbKppjeXh9MdYSGxX52kjA0zLCjfnx99J//bj9VpWmsYf8TSDHnX6K6won 152o2FFVbCQPYHjO7XTNN8AQ/sqLOBE= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=HDiojobp; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf21.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712043204; a=rsa-sha256; cv=none; b=THyadWc6mfanr/bUIwBgTvseXScZANYBH7MCFgFaEq3KkxAkmBhS5ofzKOuw2JT+mcopub LyOFRjOuh3ocuRkyE/lYu1oK+LiPL+jqOyPapymR1jGeEc71BqnwZNp1/RTNqB8CpS7AaL QflJmkqidNrkxaDI4nGLha2qtWTBh2c= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-1e0f3052145so45012505ad.2 for ; Tue, 02 Apr 2024 00:33:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712043203; x=1712648003; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YC/Kgduuu0t6Vvbwsn8PXX7BTHypb3qdZsKV5RTpqdo=; b=HDiojobpnLx3AzHXVTzpUR/FNRW6f8B9GOlwZPnEZ80MZkiriCZV/lB7YaVmQryu7m 3Nz42ILk2j+gJlXSBNlYeGxl6xs42PJCIF2hWLhHxDK8xb6/WqwcsGIYIwkaGBwvg+2R KKn7DzCESkWBI8nk7fktCWfbTIL7fj7O/a+Oz8D+GTe+T05ss13ON8TbNJZUJy/OHn2F SyVzE1xVCCDGX36A1zXjX88nfNhGcJRtHdaud0WbPKZeF3LW5wOW9dOgHlphZDI2MaK9 zfnt6kqtRtBvkVTwmCan3scVM6/iBpvnfFQkliSXNoz1RNibNaET541evgYZb2nXYfPU 7Hqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712043203; x=1712648003; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YC/Kgduuu0t6Vvbwsn8PXX7BTHypb3qdZsKV5RTpqdo=; b=Nk2GMmT0gamSiT04WJk6/v9eoWvQX5OHOfs6cBy64WZMjbCAKFyLxtX59QS7yMKuwi 38oaUYBGRQtvfOZZAdwpiaAULawiVOlHlzt43kflnQKBY/CfseQYoFuwtPmT948ysN0H cT8wgfg+RpY+LT67UbRvDfoaoEdcfD5sFVSI4QtbZIfynMRVIvi6BurN5zZEKyE5tDmU u37ta8WINi4RMvuuPZMjIGWfN5f5GrBl5IJx+MiwV6cmeI/Tykl3CIsvx2ZsYu2qkm9m cJTFG/6+P4R0eMTm8pUlDX94oaz3uREthWlwchHPWXegRBUVTxhtsj0me0/p/JjBEuTU 7Z6Q== X-Forwarded-Encrypted: i=1; AJvYcCX/38nzwg3WpIgnh7RNGKUSxFmli6nq4SV/Qayw4dweuTXsPZ+ESPGcQeLWAD1g0fXyeGEfE4YiVKkOsMFELgLMjA4= X-Gm-Message-State: AOJu0Yx4dBk94zFePXUMdgtaOzYg1Xg2v/3jbXcE3JnGmwe8LMYfskiN VlE1lMo9VKN7dKzu2weK3u3Hkz21blYnCAB30qkQDEhsTuPISb4K X-Google-Smtp-Source: AGHT+IFTa4UbxkTyNq2/JmkPohFzu+gnIk87mqQPgulDZl37xJQ6GRcUGutEVaqVEIW/xbzp7Sw1FA== X-Received: by 2002:a17:902:b20f:b0:1de:f7cf:471 with SMTP id t15-20020a170902b20f00b001def7cf0471mr11834234plr.69.1712043203122; Tue, 02 Apr 2024 00:33:23 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id x15-20020a170902ec8f00b001dc9422891esm10187588plg.30.2024.04.02.00.33.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 02 Apr 2024 00:33:22 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: david@redhat.com, willy@infradead.org, ryan.roberts@arm.com, yosryahmed@google.com, hughd@google.com, hannes@cmpxchg.org, surenb@google.com, xiang@kernel.org, yuzhao@google.com, ying.huang@intel.com, chrisl@kernel.org, kasong@tencent.com, ziy@nvidia.com, baolin.wang@linux.alibaba.com, hanchuanhua@oppo.com, Barry Song Subject: [PATCH 4/4] mm: swap: entirely map large folios found in swapcache Date: Tue, 2 Apr 2024 20:32:37 +1300 Message-Id: <20240402073237.240995-5-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240402073237.240995-1-21cnbao@gmail.com> References: <20240402073237.240995-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 2F6171C000C X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: ibpfwm7i44y371a71bdkibuub8ahb7ch X-HE-Tag: 1712043203-428739 X-HE-Meta: U2FsdGVkX1+TwqP8AOki6tOKKbbPyVI9zFqqwBj6xYsc2GgWfbQO6dGYD8zAj+41aslGnjeVDO/qcSaBD/Zl+LP0hVbDnT609Uice3h0bPTsQyD7znAO6LVGhJNQlekP4WoIKMkltL9IYOlJ16DpnDKlg2pYTGLJyN7cRk8mv2l59l6+cOkBvBnFScii6rMRK/dn8ux9q7vn0yhqEVnEWwv0Nkl+0ovzQCAb3RZu6ENMphrsvHrd0n1K6dFKkzLNA+YiIy9rHIgmhMlwkrsO07jA98OtFfIYw8zalKWxzkZw+M4YR3zqbN64zpWnc7MfKTDrVkiN1+KjlvP5voJ8zEH8T77/W8Odb4bTdvtKblaf2O24D+MbwRNzv7kYBsyM1cSFLcoHW4+/c/V5GO/8WvHR2y8ualQiBvS0juRO7PzH5DTOlT2+FrEihs7g3xYqIzyceCwuCMRkKREtrA1i+2WRi3K10bwfRFy0x9s2KIlPR2EmpD/I8m3L6tU5lgajph+FnfaEU6aI2srDCOKJRI0bw34pYt6fe1Wb3XTWOjs8cB6YlZXYWKhEJbhL4soh07QJoGG8y9oMM5v7V5xEzuJGacXW23b4ZWYfjym04iQBAV02nhrnHBLi13DN9MtMT/qjpZoYOOkKv9cDDJ9qZ/7e0STKI1aW8dlgBKMrv3LtHOaz9AUDqmuOSGTcfZbj4B49kYigqZAZaNtZ/KMFL8OMOnwaILBWNCxtC/L8OYCW7K4bAVhIDy1oGU5EgB4wCL32xSDfP8Em9jO55JFIWiC5SJJLBegfzUnVo049V8RvqO8oaBK4qerHyD03Z8DmkCBT4SNQwJ3EW02yi3ousDfnDhWeWWCQxmWhgdTZdhepjsc+y0kqU3HCllWj3MsXe7d3CE/hE7rTzqXwzbtD4dMgMMZFq2g7H+Q6ckOOq13t50QeM6HrLQDIv0txlMbvOefgd1H0cFVoMJwKuY4 zan6g7+E C8TBj1vvkchd4HNX43OHvB6LPKG6TfDr+3K7cy+q1vOtIFi8dEMf1mbH9AWybqwSf3Rt21xmKD8NIbOWEyXxfRr3LL66Dl4BATrU6lsShJ1Ugh+07sxrb7UJKYVn2AYVNuI9TBwcZKvT93Aw6778wvtqiORzYkZaQ8Jf1GgNph5zp9rKSamDuW0eyCnmEhj4gmsf/5jwhunZOqrRE2DZH0cgposLPM8AUeyk463ghu/nXCYEMjN8xvHQeiF/jXmVMc1AL/iUXKgLvnc/yNJYAf2OOdofFs9AYuMDwaXb94FW7nWIaSMAq+CggBQ9mCEtyqHEa8ofp776e9GRaI1HxFTtDX9JLpa2R1ACTwwyVmUq8jtQIkqcsFw0x4x8WxBZaj2UR/Qi9YWMv7nMpwsQIbRubij/BUM2PrqnUZ6QkpN0f8dxm+W2oblWE96JU1rwsxFrsrb6gDQ4FJ7C+XZDmXOq+rJ5n9ORpHNi9O1FL1X9BhhZuWJmUtz3z5XsZkxYn91JhjGo5il4Ih8H476awKRj06FHN35zOiszemgU9o7bsCZ58NWNlSfvoew== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han When a large folio is found in the swapcache, the current implementation requires calling do_swap_page() nr_pages times, resulting in nr_pages page faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song --- mm/memory.c | 61 ++++++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 49 insertions(+), 12 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 0a80e75af22c..5f52db6eb494 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3941,6 +3941,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) pte_t pte; vm_fault_t ret = 0; void *shadow = NULL; + int nr_pages = 1; + unsigned long start_address = vmf->address; + pte_t *start_pte = vmf->pte; + bool any_swap_shared = false; if (!pte_unmap_same(vmf)) goto out; @@ -4131,6 +4135,30 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + + /* We hit large folios in swapcache */ + if (start_pte && folio_test_large(folio) && folio_test_swapcache(folio)) { + unsigned long folio_start = vmf->address - folio_page_idx(folio, page) * PAGE_SIZE; + unsigned long folio_end = folio_start + folio_nr_pages(folio) * PAGE_SIZE; + pte_t *folio_pte = vmf->pte - folio_page_idx(folio, page); + int nr = folio_nr_pages(folio); + + if (unlikely(folio_start < max(vmf->address & PMD_MASK, vma->vm_start))) + goto check_pte; + if (unlikely(folio_end > pmd_addr_end(vmf->address, vma->vm_end))) + goto check_pte; + + if (swap_pte_batch(folio_pte, nr, folio->swap, &any_swap_shared) != nr) + goto check_pte; + + start_address = folio_start; + start_pte = folio_pte; + nr_pages = nr; + entry = folio->swap; + page = &folio->page; + } + +check_pte: if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte))) goto out_nomap; @@ -4184,6 +4212,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */ exclusive = false; } + + /* Reuse the whole large folio iff all entries are exclusive */ + if (nr_pages > 1 && any_swap_shared) + exclusive = false; } /* @@ -4198,12 +4230,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * We're already holding a reference on the page but haven't mapped it * yet. */ - swap_free(entry); + swap_free_nr(entry, nr_pages); if (should_try_to_free_swap(folio, vma, vmf->flags)) folio_free_swap(folio); - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - dec_mm_counter(vma->vm_mm, MM_SWAPENTS); + folio_ref_add(folio, nr_pages - 1); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); + add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages); + pte = mk_pte(page, vma->vm_page_prot); /* @@ -4213,33 +4247,36 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * exclusivity. */ if (!folio_test_ksm(folio) && - (exclusive || folio_ref_count(folio) == 1)) { + (exclusive || (folio_ref_count(folio) == nr_pages && + folio_nr_pages(folio) == nr_pages))) { if (vmf->flags & FAULT_FLAG_WRITE) { pte = maybe_mkwrite(pte_mkdirty(pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE; } rmap_flags |= RMAP_EXCLUSIVE; } - flush_icache_page(vma, page); + flush_icache_pages(vma, page, nr_pages); if (pte_swp_soft_dirty(vmf->orig_pte)) pte = pte_mksoft_dirty(pte); if (pte_swp_uffd_wp(vmf->orig_pte)) pte = pte_mkuffd_wp(pte); - vmf->orig_pte = pte; /* ksm created a completely new copy */ if (unlikely(folio != swapcache && swapcache)) { - folio_add_new_anon_rmap(folio, vma, vmf->address); + folio_add_new_anon_rmap(folio, vma, start_address); folio_add_lru_vma(folio, vma); + } else if (!folio_test_anon(folio)) { + folio_add_new_anon_rmap(folio, vma, start_address); } else { - folio_add_anon_rmap_pte(folio, page, vma, vmf->address, - rmap_flags); + folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, start_address, + rmap_flags); } VM_BUG_ON(!folio_test_anon(folio) || (pte_write(pte) && !PageAnonExclusive(page))); - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); - arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); + set_ptes(vma->vm_mm, start_address, start_pte, pte, nr_pages); + vmf->orig_pte = ptep_get(vmf->pte); + arch_do_swap_page(vma->vm_mm, vma, start_address, pte, pte); folio_unlock(folio); if (folio != swapcache && swapcache) { @@ -4263,7 +4300,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } /* No need to invalidate - it was non-present before */ - update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); + update_mmu_cache_range(vmf, vma, start_address, start_pte, nr_pages); unlock: if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl);