From patchwork Fri May 3 00:50:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13652245 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 054A5C4345F for ; Fri, 3 May 2024 00:50:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5611E6B0083; Thu, 2 May 2024 20:50:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 510726B0085; Thu, 2 May 2024 20:50:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3B15B6B0088; Thu, 2 May 2024 20:50:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 168D56B0083 for ; Thu, 2 May 2024 20:50:48 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 28C9616081E for ; Fri, 3 May 2024 00:50:47 +0000 (UTC) X-FDA: 82075254534.21.6E18308 Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) by imf23.hostedemail.com (Postfix) with ESMTP id 69C9314000C for ; Fri, 3 May 2024 00:50:45 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=HEJH1OKc; spf=pass (imf23.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.215.169 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714697445; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=pttzJ1oL5psmDKSRscVzWJxl+ExC61y0Qf6UalfkVjI=; b=RivMn/Ybg/OtQB5Zt1J/0AOqz2hJyoTQe9BVzkceFN+UdV5ag/GjO7ponTgpqqBEQIFii+ FLdvUnwbaFZ2dJS/ap3EhnnZYZHpTjceljEsf9hDrV8/UqLnVRMw0t0J+4keQL2LksmO26 8Fu+aJFPtf7ZKXLI6838yZSYuUrICm8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714697445; a=rsa-sha256; cv=none; b=0BXQdviyHmsvRhq7s4c59LAKeGZkA/uzA5YzCPrLCpBGnfNpgGVxFiTdD6DipSqPWUtiwV sTrrvmuvBftt8Bz3a2OIMvz0oA4j/Hv9Dubt5LewjNR/24ChYs++ZknfjLuFgBBM/OJoHe 76LwMKmQ7Hf6qfFCIWDTkapOBzQg3oc= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=HEJH1OKc; spf=pass (imf23.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.215.169 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pg1-f169.google.com with SMTP id 41be03b00d2f7-6001399f22bso5966227a12.0 for ; Thu, 02 May 2024 17:50:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1714697444; x=1715302244; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=pttzJ1oL5psmDKSRscVzWJxl+ExC61y0Qf6UalfkVjI=; b=HEJH1OKc/7eKidY4HNldDq1Sv6XPXVOa2iu1asjoJt9sTekJuM7Fv3UsCL3MkhnFy4 VgQ5R7To7weQXJ0fOOOGDLkf6VRtkIuswBuB97m2hUPU7NOFyj3THxAhMISEUa0ggzpB bPBmD2QwyO/3pveDe2WpNxw7oRyFpaRS7RpLVHyzwi38Gg9NBiXG0zbPwt/DzOYUTXBg en1L7H9YR6HDk1Va+YBOUsrjnb8Q0qiMKvnxt29x8Ec6n6oBuT663zxHK4nqOXjxhZ5I hxonIWDjMSW2w2JRezDJRnjS5ITmr9f/EJW2xwaOvXbb6cx+dgRfE1aDDOj9YAkZKahs wHQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714697444; x=1715302244; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=pttzJ1oL5psmDKSRscVzWJxl+ExC61y0Qf6UalfkVjI=; b=kPKBEvgGGayFGpTfu72tEpgi4TlOPzMxs8869j9A+2+YG7Vc4kE1M9yum1FkUGjNSK 1S4hRc0KDcPW6+MWVwr79lwlufIyJoUYagB+WE3vmVMTBhs3WgQ4vtoNjy63md/2tcLO QG4EC9XNEqIpRfskRaoe8abDeP+ipE6xRDMl2Vl3JWuSytMQgOqGI8Wl+BTfsxeOgJTF B8PCvMiz74EGBIvtP+UpxTq0q8sw0AMWrwbcN/vY8Zxq54xnjX0d+1aP86KzVyUv1Nww tBcpsLvjGyUdPPIxS8jfvmFFnjRLqDrqEG2dxoaddoTNZGOJPDcZ5gyM5muFw590USxs 8qnA== X-Forwarded-Encrypted: i=1; AJvYcCXJSug/5E5Pom8LfmHx9HxsbiVg1eZqstY+uRdfdIwuqbL/bAsEghWHt63jEJfXiUXmaxZBvnT7G8bZ+d90ZeikykU= X-Gm-Message-State: AOJu0Yx/8Vj5fk9hJKotYLxmS//v0PFLYC0O3F9ox0dXoKtqDsenEzLE yw4liFnOXqRZco01F7zBglrAfVDs6g6ydSXIomPB3dfmfukmYc53 X-Google-Smtp-Source: AGHT+IHwKWVOqhPLXUS3QgE5M8ppn6dv6AiBYd2YOzDOOa8alWeAG+3bBTD/xcy6BjQudvLnFR6p9A== X-Received: by 2002:a05:6a21:3985:b0:1a7:9adc:86de with SMTP id ad5-20020a056a21398500b001a79adc86demr1792275pzc.20.1714697444209; Thu, 02 May 2024 17:50:44 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d14-20020a170903230e00b001ec48e41d2esm1969175plh.212.2024.05.02.17.50.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 02 May 2024 17:50:43 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v3 0/6] large folios swap-in: handle refault cases first Date: Fri, 3 May 2024 12:50:17 +1200 Message-Id: <20240503005023.174597-1-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: utonson8siy1jza4cjfw741a3rbbgntm X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 69C9314000C X-HE-Tag: 1714697445-483367 X-HE-Meta: U2FsdGVkX1+WXyM3pTgD1FA3uncAb/TUYQfjTSRVlVsMx9KDlC1uijWYOE9B+G9iclR1Py+aUE1chnHEU1Gn4l6CZlr68gnpzK2EU/AdUwHrE6WHHEQw3ksHoi4SuyFfbzlIBCR3txH6ZSkeiBlDpQ0SzsPsU6eVAKa4K627Ec0kQiYGtyMHbXrxsGQNxA15n6oaX+a5wHV7by9Ub1k2xpeIpgdK5wCasbwAKy0rlAFVtwsa3TAYwQx+nlEf8MmKVHU0hoWpcVZZJah00jwB3eWrxUpK5ypKZGkaVgNwf9yL/yYyWNvERs0TLL8nD8Hex3cTvoQATb7KsJIXt0aRSHKGS+ydYWS+Y2MTwZZGcvhGtmcKHmbSbEYQLrdnv31CxyP+84oXl/VHfrFgbogzl3zCuNb0So81QyLdMSd5jEYr/ktp33vljkH/btGbIHyAUzqaCLw650bCyXyGxYnzPixI9i6437bsHc1/gWyaB4r6LQ09kDgVWlI/5j/OJFP6rjDmXjOyQwWOKI+DPQSxiwoImVkO5NTjMRuRbquSFHvAc/2BukRIXfSbuinPErrVY1wNB7+FVIMea8lk7uHeeBBJjCNUf2/o83uH42Gtr7IcrNhp2t4gT3y5rrGMSfnWb1WU9VnLtTHrNUrkAB4ZBPd5qwm3e3SDmW7GhiVx7bbdLzZlFgpGx0bzUPyFR4OhGG5yR3zxPccJwP/kXMkYWNOXIt+4Lh3U4uOy/LvaSQ97HaWBEgaBiW0iJKX8H+zC1rfAFziowZ4f2M2BuRP2XwPhhB6FdH02WhMYEoDD44cZgU2W8prf73ebcrvZFpXpirajmIfvV5kZeXomJdEbZFXcBFJZYcK/yKZZkDQ4P5+dOQadUBHIGL4n2koCpYN4B02qbKuc7GO6zGiZ3sQ26QqbyfBre7QBnwZl5WDT1sLfawcXpt9DGPnRETSBDkXha6xBg+75RKyapSw7ZL0 JVIAZQ+V wNnKLujpPQ9xCY5czo/FUDSOkrzpcqL0FoDNs8Eu5dWqYLrPJziCc4gpizhBnzr5ZWRxMENVDG6BP+O30sLw6dkb9hdQjzEBvQkJ7uZs7FkxdlgSHUmQLaAQbGvcDaDvY+v6Fm2s3xrpgXcbF7yxhfVEiar15FaZBMTdddNlG/HkUFEfCglN/8fwY5ZiCAIyORERYZ/ZrrnQGFRoFbZWcOhbbH0jtQ2pXbFbmLWali7fie3KMbdPTre+beOt9/iU6puIBDcVtqWylEhq4Z6pLv9i5oZZGmxrllkOzKdRmtQ2MRmwsDhos9qkLP4r9BVRIP56vnMrWrxC2qUh6RfMo/dcxtvtuNom9CVC/t348m6uINP+mGVCzReYZDza83pYx97wykM00jP5eAO9tkyq6juZOY4oIuJAgTXb0AJ2ll3MAfu4K+X3pcsm0zkMI68gFtIHffzu/o2HKw+XTTDmldFIy3btLK23ggCswEPFxtZUOGDkkF8qOGRkKdheF9hFzNtN8k/Hi6DJg8VW4OqehFMeu6adMw1iP4e5m3Y+d5ckDnOBkQtclMRFc2XoklY221DdDOgGBZyXkWGtbCYkrdpyQTgSWwx7HWFU2yF3/MhevJarAwFC9ozgPqQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song This patch is extracted from the large folio swapin series[1], primarily addressing the handling of scenarios involving large folios in the swap cache. Currently, it is particularly focused on addressing the refaulting of mTHP, which is still undergoing reclamation. This approach aims to streamline code review and expedite the integration of this segment into the MM tree. It relies on Ryan's swap-out series[2], leveraging the helper function swap_pte_batch() introduced by that series. Presently, do_swap_page only encounters a large folio in the swap cache before the large folio is released by vmscan. However, the code should remain equally useful once we support large folio swap-in via swapin_readahead(). This approach can effectively reduce page faults and eliminate most redundant checks and early exits for MTE restoration in recent MTE patchset[3]. The large folio swap-in for SWP_SYNCHRONOUS_IO and swapin_readahead() will be split into separate patch sets and sent at a later time. -v3: - optimize swap_free_nr using bitmap with single one "long"; "Huang, Ying" - drop swap_free() as suggested by "Huang, Ying", now hibernation can get batched; - lots of cleanup in do_swap_page() as commented by Ryan Roberts and "Huang, Ying"; - handle arch_do_swap_page() with nr pages though the only platform which needs it, sparc, doesn't support THP_SWAPOUT as suggested by "Huang, Ying"; - introduce pte_move_swp_offset() as suggested by "Huang, Ying"; - drop the "any_shared" of checking swap entries with respect to David's comment; - drop the counter of swapin_refault and keep it for debug purpose per Ying - collect reviewed-by tags -v2: - rebase on top of mm-unstable in which Ryan's swap_pte_batch() has changed a lot. - remove folio_add_new_anon_rmap() for !folio_test_anon() as currently large folios are always anon(refault). - add mTHP swpin refault counters Link: https://lore.kernel.org/linux-mm/20240409082631.187483-1-21cnbao@gmail.com/ -v1: Link: https://lore.kernel.org/linux-mm/20240402073237.240995-1-21cnbao@gmail.com/ Differences with the original large folios swap-in series - collect r-o-b, acked; - rename swap_nr_free to swap_free_nr, according to Ryan; - limit the maximum kernel stack usage for swap_free_nr, Ryan; - add output argument in swap_pte_batch to expose if all entries are exclusive - many clean refinements, handle the corner case folio's virtual addr might not be naturally aligned [1] https://lore.kernel.org/linux-mm/20240304081348.197341-1-21cnbao@gmail.com/ [2] https://lore.kernel.org/linux-mm/20240408183946.2991168-1-ryan.roberts@arm.com/ [3] https://lore.kernel.org/linux-mm/20240322114136.61386-1-21cnbao@gmail.com/ Barry Song (3): mm: remove swap_free() and always use swap_free_nr() mm: introduce pte_move_swp_offset() helper which can move offset bidirectionally mm: introduce arch_do_swap_page_nr() which allows restore metadata for nr pages Chuanhua Han (3): mm: swap: introduce swap_free_nr() for batched swap_free() mm: swap: make should_try_to_free_swap() support large-folio mm: swap: entirely map large folios found in swapcache include/linux/pgtable.h | 26 ++++++++++++++---- include/linux/swap.h | 4 +-- kernel/power/swap.c | 7 ++--- mm/internal.h | 25 ++++++++++++++--- mm/memory.c | 61 +++++++++++++++++++++++++++++++++-------- mm/rmap.c | 4 +-- mm/shmem.c | 4 +-- mm/swapfile.c | 50 +++++++++++++++++++++++++++++---- 8 files changed, 143 insertions(+), 38 deletions(-)