From patchwork Wed May 29 08:28:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13678361 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 503FEC25B7C for ; Wed, 29 May 2024 08:28:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 80DC76B0099; Wed, 29 May 2024 04:28:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 796966B009A; Wed, 29 May 2024 04:28:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 60F856B009B; Wed, 29 May 2024 04:28:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3EF146B0099 for ; Wed, 29 May 2024 04:28:47 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id ED299140CE9 for ; Wed, 29 May 2024 08:28:46 +0000 (UTC) X-FDA: 82170757452.28.4C673F1 Received: from mail-oa1-f41.google.com (mail-oa1-f41.google.com [209.85.160.41]) by imf22.hostedemail.com (Postfix) with ESMTP id 3317DC0002 for ; Wed, 29 May 2024 08:28:45 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=W9DKymjb; spf=pass (imf22.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.160.41 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1716971325; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=ewOefsnjLcZF9k8X2fk5+VmjOrQKwuSB4YjvYAvVnjc=; b=ZAWoS39inTsjHFWAs/7yXBo4OGitkYgNyN3DxrgKQ1SeTOQkRvvzG8XxHj4j+QsF6J+PLn TOd4ZyuD6AyRNjItn+OcGfMjKsSe6ghBTCwMudG2FBDCIfVjTlmmcsLyoovE39LZPsqno4 6bkTEG5tTj3AH3IFiAjyc5jr2Yxarew= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=W9DKymjb; spf=pass (imf22.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.160.41 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1716971325; a=rsa-sha256; cv=none; b=eXo64YSMNYRaCsORD7oygb6zhjspXlsWy12DaFS6RjSY7qbGtcjnJxFPDMVRsxg2iZofud JJxmQVi/3z3cM5MbtEfHBPEV491vAMS6pTfWsFcuFr0qkVv08XieqNMpsUy4fdkyZ1kOrf sl1tavoi12xtvmBuPnfa7PGE85roUCU= Received: by mail-oa1-f41.google.com with SMTP id 586e51a60fabf-24cbb884377so843131fac.0 for ; Wed, 29 May 2024 01:28:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716971324; x=1717576124; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=ewOefsnjLcZF9k8X2fk5+VmjOrQKwuSB4YjvYAvVnjc=; b=W9DKymjbzRh8I+3a9q0Hk5wIXvnDYA2FW+JMgjvAMSF7ANRVuPH5/OBsB9D2cTfPP8 BdJCjCQyG06l4neAGIcQpiT6way6Kcgrf1iFgnzb0O8ex+1yr5urh55M8yU84v/o/+ni mOHY24fYVquYMC23QiIpB3lTR3LLWE4uX2Mtze/b724UyDyZ72YqxYWbOXpOZOGJA6w5 5u5AM0sxBh2LvQU9oTuu/vK+9taY69fba4xQ/M1jGBnzjH83hN9eXkOsbssMRGkI/f4v cypftMFS4C1m6tH9FcJp4d3O/uorIzss1MNnOTRQpWZvP/Yu46nitM5ImGfGn/uJwFOk Wtrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716971324; x=1717576124; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ewOefsnjLcZF9k8X2fk5+VmjOrQKwuSB4YjvYAvVnjc=; b=ktjY+IG6INn0xOSnQwZnGTAMDqcwI1Tc2sWEnGfkO0wz4vkENe4wELbKMw4HF+O5w3 0lPBK9asiZmZEQFWKlxh59z2EwHzm0y+MlaTT5HPQKrJ1r8HwB3X7lhl+G7rK3xCdAUm H6tL3K+iYO7uax14uHOuTFGbsvCBBdAT04E9Sn3KmHLaJZPozJyYR6mQIKFPrdi13ZAl lgiGJ7serbvIX+CQIjrdnGAjgT8Ulr23D5Eqf4eugyD2HVv28fFzfC4XOdDvzf3rl+Gj tZt+ch0YLMr/mNWEYNFsf68Ui8+pFGSg6aE0aqUl/ydkXgMEHwbCmwR3hXJbW+XNwouP Hs9g== X-Forwarded-Encrypted: i=1; AJvYcCVC8nhWMN956s7XF/aYoVp/sTtVcVBsWzSWwvlgv44f47KUNoNSGcQ3B0xYQNb0dKsGZYcHYycsQproRYcJb9ulTFM= X-Gm-Message-State: AOJu0Yz3n/FoY3h3xsrFbJoLsreNrdHpWc80CsmtU1YmkFHUyNg8fZnf P2dZWp41sY9zlJWgwOyFlUs2UFGRucZ4bp09CtWDEpEsO4cRIiXV X-Google-Smtp-Source: AGHT+IH3UNmleoitLaaWKEkGkztsqlk0g2xMkG4KbUUl9w4zYCJXpK0DOsfwuTA5F3olq/PwYIKyfA== X-Received: by 2002:a05:6870:4722:b0:24e:4fef:61df with SMTP id 586e51a60fabf-24e4fef737fmr12412326fac.54.1716971324013; Wed, 29 May 2024 01:28:44 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-702220d948bsm174916b3a.131.2024.05.29.01.28.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 May 2024 01:28:43 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v5 0/6] large folios swap-in: handle refault cases first Date: Wed, 29 May 2024 20:28:18 +1200 Message-Id: <20240529082824.150954-1-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Stat-Signature: w1kw37k6nt7ctxswiscnsk1zmko7eptr X-Rspamd-Queue-Id: 3317DC0002 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1716971325-581087 X-HE-Meta: U2FsdGVkX18gCFrWc9t6obrL0nFGFUBHZjr+44Mxeea/ZsdHm1FCkECBPgVWpPpPR2WjSUSJ6ZRmBEss6fmHlJ8d5u+Ei+Sf2m7kvEwduNbA/w/Qmg3JmFbko2WcUFLzxXC1CBdycUWZmEdGOPSuEVjhlSWKWiyGYa0V3kiw5RtfRx+CKmZ0pfX38AKKo7vroIOppWko6nlNDF8Yt64pKhmgauJbQQBToFVvRf0r/BakAfoRu/eok7R1ujPIWcfx6svz5cD8xW3NVoCe1VrU/nC+E308cXWRS5UH73Dq7WB89AGWvWJYdn0NkxpV/aCK3EcreWpysQXzssEPfwGxw/fMdhigaVRAa3uGr+brKtWxC5FD9quScODeeG02lllD/406qHbhmMZ4HJEPqIb5/54C6y5OoXLmD75z4JUvG7V3j+gSxURxGmodN5bgfiJJHsIYoSkIyF18ucQbaYOBxIrUuWwSkxnlpc9k/jibHvwy7z2njPmnOpHZAlV6dEMtp1bNBt2E23vC3eRn6E9Y7NxBgqmr7AKQ47gCewdIspZx+AtvPEWnojoYr0OR37YZBVOWcbMDbTNCAxr2IwUz4XaPljlQWKH33UiL/ZKorELHKqsbj7NPLCJlZRXfLSeqdBYnnWwf1xCe/toozRm9Q9FHOB1CiS+EYCXmO7YC/ExhnUSOkkZCJcGsqs+8Fx/4mZk3CoET//FRE4Ivn9+XQsvYc1XPN5croBHreZ6GcFCV+QuS48IqVipt8yk/3L981Ze0CRelVkLtdSZs4ZCSKn0VtJwC537Fo/dHCrgpUFIL3AGTYtChs4IZY+rzYARirIClPE+UBDSJqizZVBAOoNOH9Ajaoz706if7g+dT0dI8XiMB2BS6S73+ukPFgQLH3basQajOTyBLhbXtDng8jjwnJHYxBOluz6x1qDDa4KWuPgizteEet9wGFcDxKVD+rALzvIi+wTnRvtGgm5o eGMq/F7T 50nQUHgUUeEUnUjX+hwaqTQLgavYS5pJiUiyrdPglDgNDNE5CPcr6NwIDxOeb2N6uVzfp8vDI3eSDFBP5WUyoo70ppD+6t6x0G6KB0gCZWte+IP8kRhJ561nHcdW/NGgQsWBdQYaN1cGepqMWfH/Sl5lBJpd4WjiZ7seRnWjDQgsrKSmclx1Lw5XQ8XE7x/CLbhL+cT+KwegFrRRcw40xZKrZwkYuZ9upZzYSAle4Y3kkhm4JxMBbjuyLPK317JzWAc47OAOMR+2CxJqI26rFGbnqRCsEkwsEwymqtCECsKYS+k+IiuqF3JWFcHUc2jkq2N3lljkDW006IImUSOgROhHlXEFlS2uPq2GJ2o/5uPokvR3NZi3S+12GbBU657g/81jCBEsw4nwxbleOFNiLmYDRd+oflJCLfliLieXY8PpHn41TMrcXWrsV2M1F3jjESHBBGXDhptv25fN1JuXitygqeoiA+Wwhpo+TSz+oowoRQVxcjx2VomxeHc/pQEObd2DPuAb3n/5SruYqzJPh95rgl7WXoDT5QkNaGbFcEmt9fgxm1Hj9BklLlVmz9/pzeTDjB58TxRTAdrJqOTM8xq3CIu5fS2NKmAynNQD8IxVoQnFwyfoO1hkyHneptNE7pIhD X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song This patch is extracted from the large folio swapin series[1], primarily addressing the handling of scenarios involving large folios in the swap cache. Currently, it is particularly focused on addressing the refaulting of mTHP, which is still undergoing reclamation. This approach aims to streamline code review and expedite the integration of this segment into the MM tree. It relies on Ryan's swap-out series[2], leveraging the helper function swap_pte_batch() introduced by that series. Presently, do_swap_page only encounters a large folio in the swap cache before the large folio is released by vmscan. However, the code should remain equally useful once we support large folio swap-in via swapin_readahead(). This approach can effectively reduce page faults and eliminate most redundant checks and early exits for MTE restoration in recent MTE patchset[3]. The large folio swap-in for SWP_SYNCHRONOUS_IO and swapin_readahead() will be split into separate patch sets and sent at a later time. -v5: collect reviewed-by of Ryan, "Huang, Ying", thanks! -v4: - collect acked-by/reviewed-by of Ryan, "Huang, Ying", Chris, David and Khalid, many thanks! - Simplify reuse code in do_swap_page() by checking refcount==1, per David; - Initialize large folio-related variables later in do_swap_page(), per Ryan; - define swap_free() as swap_free_nr(1) per Ying and Ryan. -v3: - optimize swap_free_nr using bitmap with single one "long"; "Huang, Ying" - drop swap_free() as suggested by "Huang, Ying", now hibernation can get batched; - lots of cleanup in do_swap_page() as commented by Ryan Roberts and "Huang, Ying"; - handle arch_do_swap_page() with nr pages though the only platform which needs it, sparc, doesn't support THP_SWAPOUT as suggested by "Huang, Ying"; - introduce pte_move_swp_offset() as suggested by "Huang, Ying"; - drop the "any_shared" of checking swap entries with respect to David's comment; - drop the counter of swapin_refault and keep it for debug purpose per Ying - collect reviewed-by tags Link: https://lore.kernel.org/linux-mm/20240503005023.174597-1-21cnbao@gmail.com/ -v2: - rebase on top of mm-unstable in which Ryan's swap_pte_batch() has changed a lot. - remove folio_add_new_anon_rmap() for !folio_test_anon() as currently large folios are always anon(refault). - add mTHP swpin refault counters Link: https://lore.kernel.org/linux-mm/20240409082631.187483-1-21cnbao@gmail.com/ -v1: Link: https://lore.kernel.org/linux-mm/20240402073237.240995-1-21cnbao@gmail.com/ Differences with the original large folios swap-in series - collect r-o-b, acked; - rename swap_nr_free to swap_free_nr, according to Ryan; - limit the maximum kernel stack usage for swap_free_nr, Ryan; - add output argument in swap_pte_batch to expose if all entries are exclusive - many clean refinements, handle the corner case folio's virtual addr might not be naturally aligned [1] https://lore.kernel.org/linux-mm/20240304081348.197341-1-21cnbao@gmail.com/ [2] https://lore.kernel.org/linux-mm/20240408183946.2991168-1-ryan.roberts@arm.com/ [3] https://lore.kernel.org/linux-mm/20240322114136.61386-1-21cnbao@gmail.com/ Barry Song (3): mm: remove the implementation of swap_free() and always use swap_free_nr() mm: introduce pte_move_swp_offset() helper which can move offset bidirectionally mm: introduce arch_do_swap_page_nr() which allows restore metadata for nr pages Chuanhua Han (3): mm: swap: introduce swap_free_nr() for batched swap_free() mm: swap: make should_try_to_free_swap() support large-folio mm: swap: entirely map large folios found in swapcache include/linux/pgtable.h | 26 +++++++++++++----- include/linux/swap.h | 9 +++++-- kernel/power/swap.c | 5 ++-- mm/internal.h | 25 ++++++++++++++--- mm/memory.c | 60 +++++++++++++++++++++++++++++++++-------- mm/swapfile.c | 48 +++++++++++++++++++++++++++++---- 6 files changed, 142 insertions(+), 31 deletions(-)