From patchwork Tue Apr 9 08:26:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13621993 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52948C67861 for ; Tue, 9 Apr 2024 08:27:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D7EE26B0095; Tue, 9 Apr 2024 04:26:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D2F316B0096; Tue, 9 Apr 2024 04:26:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF9DC6B0098; Tue, 9 Apr 2024 04:26:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 9F94B6B0095 for ; Tue, 9 Apr 2024 04:26:59 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 5C7A91A02D1 for ; Tue, 9 Apr 2024 08:26:59 +0000 (UTC) X-FDA: 81989312958.20.D3C02CA Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by imf18.hostedemail.com (Postfix) with ESMTP id 88C801C000A for ; Tue, 9 Apr 2024 08:26:57 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=P9ORVcsi; spf=pass (imf18.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712651217; a=rsa-sha256; cv=none; b=WtuOc03Ze7YN7QNQaQv3Dwg+5joL60esj/8o/fMdjH3V0Z0wxWfHcTgSHUxxoRRxgIay4l 7TuZutOpDuo+bA8ilo4Bf/zmFvSUUn8WLnle3vWLkkjhg0U4K8f0x8rBkE9r4qp/XUxiyL WkB4wAJ9bEHqti3oWqiG0ECVh0fQrDY= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=P9ORVcsi; spf=pass (imf18.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712651217; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ei1Crh3xKYkUssDLy2Y3ak8IjHOYPwPhmj2M8HC/g7Q=; b=Fgwc3HZum97w8vE+n3eqf/8ZJx0JPb/UOpQmICr99z5xYzFSf35smqx2LLghCzYNBrCigr s0J4Ur8SAyixcPMVtti+cRzv1W4UbYHSoazdwvIQG2g/qC1W0i1Dc0PC/Yi7YOUAXvns4W 6KIJbRDIewcxF0p2ZBJohGLV9fN2H40= Received: by mail-pf1-f179.google.com with SMTP id d2e1a72fcca58-6ecfeefe94cso2614595b3a.0 for ; Tue, 09 Apr 2024 01:26:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712651216; x=1713256016; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ei1Crh3xKYkUssDLy2Y3ak8IjHOYPwPhmj2M8HC/g7Q=; b=P9ORVcsiLL32THHaYzbIHOK2l+jQDjlDtxWAcvM/LT3OYt00o8gIKejUW4JdiWQdVZ 1qu6HAqni+EkATX24F9Ov0IrjpLiV1AlcrebrEs96JH8dFTf/GzqCIr756Gm6UU2jntn 8DfZREeMGXGzN1kEUqjQzEY7G0TrmPCtzu7ZEmonhjJFbuGxJZpAOLD1KM/uY6WNkMHX u5g9N9Stvz0tZb/S8J/WNeT/M0gUvTE9omeVgEl00nmwCDWU/y+JDWdPas9QE3RtCeVn r63vayyGqCdIIdWRFFGXMAYNcO1MIlvJYG64gQrtzrJ4KPgPeTR1vqKPPj95dV8V856b 6LBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712651216; x=1713256016; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ei1Crh3xKYkUssDLy2Y3ak8IjHOYPwPhmj2M8HC/g7Q=; b=SDmZNXBzG+zzhFk+FLwTe2L0VlXKaptUNqZbenWjmSIIhQ7n3+OPf0ZnmzZGM6BNh7 lMHU62VerlDxcWf7ha9SSOv47MH4z9MeVjOgti/vIAG4kvutr9ib3ji72NifzYlIRnGu 2aJwEHVGEVoESqscDZuogJBk7H1x3hQ5caw7PT/KdafbelbwR5Bacx2hNrqrhNRzBt2k 7NCzJq7Dw7uAdvx1y05HsPJvS0JEs3XbttNe5spIYVxW0XcwuNww8OFbF/VPu/cG4+y/ /r2gjql0XJM/yDTeBzTaAO2PXC4yZmFJsEn4QwnIy52jM2Iuz2eQMlknnY5TUV5TGqqf hrIw== X-Forwarded-Encrypted: i=1; AJvYcCXpm0NHtmLKhJBeI0jA3N1ukiFeoO08gRpqrP75zt5to+N0nfgr10PLcxuUSZSdxq7fpF9DtVfG67gFLg9PaG563ZM= X-Gm-Message-State: AOJu0Yz3n2AuFcR/qnSLPoD4mOjF/z/YmqOl0jPMUZsLDONdsldVQX0G +/IoUjCoCND8MM7W8JPxkuCy+QSfjt18WiXQ5LIb9mjl0WxY5OWc X-Google-Smtp-Source: AGHT+IHZGO/8Oq5G4AI8d7uAa0Xfd/UfJnN/mN1F1Zgkmt5jIyqoC0gKP++Z8vrGbui309CoYrBrbQ== X-Received: by 2002:a05:6a20:430c:b0:1a7:5413:1e6d with SMTP id h12-20020a056a20430c00b001a754131e6dmr3319733pzk.4.1712651216427; Tue, 09 Apr 2024 01:26:56 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id r16-20020a632050000000b005f05c88c149sm7594238pgm.71.2024.04.09.01.26.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Apr 2024 01:26:56 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 1/5] mm: swap: introduce swap_free_nr() for batched swap_free() Date: Tue, 9 Apr 2024 20:26:27 +1200 Message-Id: <20240409082631.187483-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240409082631.187483-1-21cnbao@gmail.com> References: <20240409082631.187483-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 88C801C000A X-Stat-Signature: zuncr9a7waehher8efjbjf7om6xdf9ak X-Rspam-User: X-HE-Tag: 1712651217-247917 X-HE-Meta: U2FsdGVkX1/icBn/OCkG8/xK+XNaPWuROp5mXRu7YYg2glnwuuZQadfb5RKW5BUGWNPo3K9GT+fMe7HCTzHhDAOvoE+NreJHYwvxVqYXNX0YUKE3XXwhE+oWnEbsIUcOYU6z7TO1cuieQ2IbBTx9ZnGbMBSWZ+MWj1h5QcJYcOklTGyQwJtGsj1Qhhb4FkA975yP33YlU9LStyHybBJV4epYpzGXERTFpU8/2Hzp9gMmnqO6pxuNqINWtljVns4NkIHSsKM+JL0trq6upkhiMLFoXMMeUQZCBoKMg9Fde2xCtvKI8oILtIej+fKLpNQfVT6nsKs189X2lKGWJVsZdi3d6M54pJWMDhC7+qv8XV9AOx6U4LWEwp9DdSJMnWzfbvMB7wYDaEPLq+Bx6czDcjmvlz2n4xmycoSwm/1IFOD0Tpd+6NFTjxX+jjZLxT1JTj3NMg9w8Dl9oU0u7eC2x1cI/p14KICMlI0B9g7DFkC2ySCT4R50STIGmms6qvFrQt8DLQ5THa/sKwNHfEEFWYlUFmPev6U2zW6iR19pUIO/964KNGWkucd7Kt+viaxbt9LUHnuLw8RIAJvPlBmQJ+zUz2dYCw4UAXJBFKZH/5J2BIVzhY+dNUw8+5Bae3ylfXhAu/vVVV2+/4VTCbJyReO9pm19WmUHCwc8JeMOKObD9Jt/eIRTDAoCb3FjarsQWs4eXXeV384Om/8TNfsGicc10okOr5nfvOMXVJB4uOqHOqOaWRHut76wE/vAHX994uw3pwCLQdfgkOHpmwj3WQm730/zILWK4zOspwlzI5INFHUiHRG05Iheu1UmTQQYjW4mcCIzkxK3ArwHWEdoHOWl0H3AF2KHF6fqDnzuyceksXVOlJ+IRUree3SJh7yxCFmUqBjdUPW2KAGJPGA59+ru9V0L/NHs+2W6ocoGyrlHjciqM0Ge44L8dgCKP4IDLdFeE8/B3I7h/10gF7J sqJgUaoU I0g9zRECxqWBu7Toa0A7scEDtbsv174rNyZNdwwWDXqKiRRpZdhMNpjCbO3r8fiMX0CSo4hrJ1y3dT+/kqBdPRjWhGFPmiENeL2JCw2VQrxTaO6xr2ahI8C13Afly7M0qz7jOTyrFZ63JVuPBatZhROWTbWJG10X4PksBt4Br0dHDyJAkIRyz/EQBqfsLd96cfffEJtz73hedp7/mt9F6u7kdCUcgNwl8S+Pn9404Z8S84wyOiMQqJ1uinn+7Fi+YC/65k13S4xu1yh4KdpmMRmvHRzvLL1WBuxox0ItrE1eaPdMTA8J8dJAGZvGt9E8rTOBb9Cvh14qy/G6HVr3hMe4whPeqNlHYClL6nrTYILSHrpd/kFUxLg+rg6cbQ+3WBdy+mJomT9E+vBIXd6rtPUohU61YG61e8ihkIfx5T3VpfkFjv21jbxUnCzScPgrga1qX/Am8CIiK+dNWMaNlfMdzoX4Z0K99KxfABK0hXbITUuwOG6SK8Bi8Y4BmXOeDINobOegtURPIlT5/HeS7+4MDU5zhWDm24rVWgoa6gIE1MtYTAW58HwFbOiFD64TN4phG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han While swapping in a large folio, we need to free swaps related to the whole folio. To avoid frequently acquiring and releasing swap locks, it is better to introduce an API for batched free. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Reviewed-by: Ryan Roberts --- include/linux/swap.h | 5 +++++ mm/swapfile.c | 51 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+) diff --git a/include/linux/swap.h b/include/linux/swap.h index 11c53692f65f..b7a107e983b8 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -483,6 +483,7 @@ extern void swap_shmem_alloc(swp_entry_t); extern int swap_duplicate(swp_entry_t); extern int swapcache_prepare(swp_entry_t); extern void swap_free(swp_entry_t); +extern void swap_free_nr(swp_entry_t entry, int nr_pages); extern void swapcache_free_entries(swp_entry_t *entries, int n); extern void free_swap_and_cache_nr(swp_entry_t entry, int nr); int swap_type_of(dev_t device, sector_t offset); @@ -564,6 +565,10 @@ static inline void swap_free(swp_entry_t swp) { } +void swap_free_nr(swp_entry_t entry, int nr_pages) +{ +} + static inline void put_swap_folio(struct folio *folio, swp_entry_t swp) { } diff --git a/mm/swapfile.c b/mm/swapfile.c index 28642c188c93..f4c65aeb088d 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1356,6 +1356,57 @@ void swap_free(swp_entry_t entry) __swap_entry_free(p, entry); } +/* + * Free up the maximum number of swap entries at once to limit the + * maximum kernel stack usage. + */ +#define SWAP_BATCH_NR (SWAPFILE_CLUSTER > 512 ? 512 : SWAPFILE_CLUSTER) + +/* + * Called after swapping in a large folio, batched free swap entries + * for this large folio, entry should be for the first subpage and + * its offset is aligned with nr_pages + */ +void swap_free_nr(swp_entry_t entry, int nr_pages) +{ + int i, j; + struct swap_cluster_info *ci; + struct swap_info_struct *p; + unsigned int type = swp_type(entry); + unsigned long offset = swp_offset(entry); + int batch_nr, remain_nr; + DECLARE_BITMAP(usage, SWAP_BATCH_NR) = { 0 }; + + /* all swap entries are within a cluster for mTHP */ + VM_BUG_ON(offset % SWAPFILE_CLUSTER + nr_pages > SWAPFILE_CLUSTER); + + if (nr_pages == 1) { + swap_free(entry); + return; + } + + remain_nr = nr_pages; + p = _swap_info_get(entry); + if (p) { + for (i = 0; i < nr_pages; i += batch_nr) { + batch_nr = min_t(int, SWAP_BATCH_NR, remain_nr); + + ci = lock_cluster_or_swap_info(p, offset); + for (j = 0; j < batch_nr; j++) { + if (__swap_entry_free_locked(p, offset + i * SWAP_BATCH_NR + j, 1)) + __bitmap_set(usage, j, 1); + } + unlock_cluster_or_swap_info(p, ci); + + for_each_clear_bit(j, usage, batch_nr) + free_swap_slot(swp_entry(type, offset + i * SWAP_BATCH_NR + j)); + + bitmap_clear(usage, 0, SWAP_BATCH_NR); + remain_nr -= batch_nr; + } + } +} + /* * Called after dropping swapcache to decrease refcnt to swap entries. */ From patchwork Tue Apr 9 08:26:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13621994 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7908BC67861 for ; Tue, 9 Apr 2024 08:27:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 086826B0098; Tue, 9 Apr 2024 04:27:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 036886B0099; Tue, 9 Apr 2024 04:27:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E194A6B009A; Tue, 9 Apr 2024 04:27:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C0E856B0098 for ; Tue, 9 Apr 2024 04:27:06 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 895FBA030E for ; Tue, 9 Apr 2024 08:27:06 +0000 (UTC) X-FDA: 81989313252.07.A3AC693 Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by imf01.hostedemail.com (Postfix) with ESMTP id A98E540010 for ; Tue, 9 Apr 2024 08:27:04 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=fgmqMxBP; spf=pass (imf01.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712651224; a=rsa-sha256; cv=none; b=vC/oOBUZf6VBUn0auly21xq0/P05KEsG5Ym2v0kg7EYlTZiyTuX6kAFQn44XRk0oCyMEHU pOebRFV3LvFV5/cIsDM3E0GGRNiSozmdSPQt7nhhnOBHjdm3KlUVmOk4dJxEMMKC9fXBJd AYu4C9u+gp/NIQu9/cdh0Kx2bXD4WcU= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=fgmqMxBP; spf=pass (imf01.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712651224; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GpjEhAQtgqO+DLm71KCeo2y5Y8KyQFcwcH3a5IWajwE=; b=p9sUQBOWV5YeW0WoxZ2gGRZig+rg1+r/TQbce7Jgk2hqadVM69Xyn/freF50bm3hfUaeRA RRurlhWhiPz/PpnUVx/pONYYkUwYO9XEbz+x4GAelVFFXuwdBsTdnpEN75TW+TlTlC7cEa A+86HnoyOs4REp6p5Be/zpw+Socvp/0= Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-6ed267f2936so1561891b3a.3 for ; Tue, 09 Apr 2024 01:27:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712651223; x=1713256023; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GpjEhAQtgqO+DLm71KCeo2y5Y8KyQFcwcH3a5IWajwE=; b=fgmqMxBP9VH58E7K/aX7jpuylVfP/I7ioy0txZPMp0VuAnce5A00usFa4Voq4EGntl a+XoXnipzhvp13K1Zp7cCo5CpwBULF8LQvqnpuHWyDonPR0Ig1w8Z3kVpE/+/aei/aO8 yI0u4WD32ZXKIs1egM4oaAnnP/jYyWArMNunCpzmjUgnNYBl+Tnye+DekP1C95XNc3iq GtlrxVrmV8BeAL8amit80Ot05qp8EMx3MQPkg2LmjjwqHVGibM5YkPgYv/2/Wl2WXprC I3V5Ja7yrQzWsB0g3dVWQviunS9b4BW9w5LZKqn6zIkMfv6Hmp3Js5woBbOB28H04Crv eWtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712651223; x=1713256023; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GpjEhAQtgqO+DLm71KCeo2y5Y8KyQFcwcH3a5IWajwE=; b=DaaXHmK+iHSs7En5RoRPHtA0O1XmW75nXkhqYZdvupXUczDp9wMmdBl+SQIOXTFco9 gE+Xq/ISX2DIrX1emeVWfA7FS09Ol+z191Rc8gRxETZcrnbQMnQAchLJulsHM3wBHnRL ziTR5zMsbQLrBApBX5Ct3MNfVxBpKmAChyINa9WwCSkCFb+PIQMNtTRfAyaOO8S7bAvA gyKJ1gX0NEVyCCxfPY1hCGBQZSIYsYvSdnhoOVCrmcnkAFE1QibauKRK8TmmhLAtGOpG nKDa2fEwy5m/qC5MxqkhhuvASKjGYQxlCXwKsyLH0Z4DBiS0UJCvHv2VZlWBu//qTGa8 O6JA== X-Forwarded-Encrypted: i=1; AJvYcCXB43pS7tFb5GYg5r/7glKRRSpbDou3UZ6G9a5BtFPWbjGSIf6TSw+uK5U7DPQ0FJ/HdyMxdi2vDFrToGnQq5scjgw= X-Gm-Message-State: AOJu0YwCDXIKUhXOAQRfuuWlEtJryB8+GKK9VAhOlp1DwtqlF0gyKYPi MhTVFw6y97H8HIY1m4pHIvtRp0F8RKLUO5R2V+4ab8d8vze2MAUj X-Google-Smtp-Source: AGHT+IGLXAZ+gunF7hyCXzEaTPUaO18uUbecCeYC/lMFr9OrqEyLkgxoorBwDY+pMLEp+CyQ1vQJpw== X-Received: by 2002:a05:6a00:4f84:b0:6ea:afd1:90e6 with SMTP id ld4-20020a056a004f8400b006eaafd190e6mr11705280pfb.6.1712651223528; Tue, 09 Apr 2024 01:27:03 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id r16-20020a632050000000b005f05c88c149sm7594238pgm.71.2024.04.09.01.26.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Apr 2024 01:27:02 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 2/5] mm: swap: make should_try_to_free_swap() support large-folio Date: Tue, 9 Apr 2024 20:26:28 +1200 Message-Id: <20240409082631.187483-3-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240409082631.187483-1-21cnbao@gmail.com> References: <20240409082631.187483-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: A98E540010 X-Stat-Signature: 4nb1rnuw4kjfcy5hkzg9ihzeexib9jrh X-Rspam-User: X-HE-Tag: 1712651224-563437 X-HE-Meta: U2FsdGVkX19unyeTArdUvEl8BJFHL8XqcCJjYnRp7ICHpREsmUaqWkybIb0Va9FqrzRj0/ThYtF6IOCTuxs0CrpuugoYA8bjS3yfIx+3Ap9sa6iImkdBYs2ZX+SesJhL/euAj9VoW8++9OkgObEaGwfUfQfcO6FfKRm26TgZbs5MUBdXrnvMGLMrZNaG1U5YL8KGQpO10n1NlwcwKQ/aqNt7Jx2O5H+FJ3bPQSbFOUQGXd+WohvYULyuryX45iR9Z68ZMaqCndMrLXE9dQNGsqv3mcGKE/cMzV8mkVOixfbTBPV8rChO6kRvWUes6u6acVzOZzRkx4M83g66pNICdnqJ9SDJI4bkHXY4KshyFmJN04h+x6ul9DlGCBJggpdDJGbM1U41Ycwws0Kcwq8SsqkFTdPwhtXaFXGAHXpATrmoXGwUbFchOSfJOMXL0x6W2WrFz86fXEeCnmuLEsmzTcoDfkgY96DZtPtQYQOepr9tcTaGNokOJLF60Ov7QOdxIF8aJNaLrxGtrykzX6AA3Z1EA8P5zroxp5fwHIXNHD6nCPgrOebYlUYZvSKWga+Q65KZU1iBrObfg7OWS6Vgm7vJ+n2COysjH7uTCOY1rP5gID5T1a0SEitZteHjZyLRm0Xp3AhTqXWhJ8JiGGyye6hoN4fhKiNHKCN+8RGIlLzrEpkZ4TEKAxf4FVCAx6Ler0xBxlrHP87HALA+ecEbJ5mC3vO6cMqulEjgUgoZ1WrRupFj6n/lejl/UOAdEZoypYrCoSWmNdyafZvUN9y/oAwEaSa7mUZiSIDGfeBQvzs1aknmII6AfG4lflfKT4fT0qcdqgbo29Zxy4AIR/rbjZNe42ZdnQ029kLrB2vgV4HIn2FZ1L7aqFHdujnUxCQBJnHG1U8y+xbgBibh06Ts4PzCEF9zEQoDG8BVTnJtGb7huSK0cszyl2EwKBXbUuoj/TaCUZ+uyMdDprp65F1 nPIfS69h 6Q7ROp0XIoTx9xuOR7Q980XXdKG7EjGnixAeAmxlOzR2RPQQUGLrg1s31xsHK2PyDh3k7ymopEqVe8g7TdpNPonWOIXCJBni5D3hdfq6Ct0X4mklSuSiFO8bfF3wg18ShQB58LhyKsVQvBe702pVdtV2E2htx2Xw1OJcEDfZ0A5jwT9PTTkmcBYvuIHbVijbLfoqSMKqNeyKdtwPnepyQXdzJOy7dKYPXfm+60DRCO5Nq4rrhcX726FnCTUSDrPA7/N3I/Zf0gkojh6cYEYtAeZlEceGlMih0VzoPuT7NoBHU5mkRSi1w90dljWwabP/SzAu73t0Go6sRV1NuNh+D9GHQhWpYR0oabTgD99F7tpIEFOcUhSnE/jc/Sp1lbQaoAKlTqj4ljaVBHbc9vcAcA6RWNj+nuPUz9DWa/LAt5R5OAtyYa4uRHqpe9aDg3dAETWWqf+gmsBOa5QaRDclQeogBuIvlFFVrjyFnZ6zCQpkZ2QfVrRugGfyJNye9Ld/29WQADMM4HFb7P42tU9i2ERgKppupgIiLH8Ory6BTn+LBbLlNYqQvuv4+KtX+ida6QHXvvsDBf+mkw3YlnCzcT8/Tv8OJ8uMHu8VLpC30dM4NRKjI9T3uZu3U7lBD2t9/DRQC/nBdUdW7tBVegRjUIvdX1EAXnXYCeCB/tpsE2ExSmB/v4JXtSgjL9weiZZfFpE6I X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han The function should_try_to_free_swap() operates under the assumption that swap-in always occurs at the normal page granularity, i.e., folio_nr_pages = 1. However, in reality, for large folios, add_to_swap_cache() will invoke folio_ref_add(folio, nr). To accommodate large folio swap-in, this patch eliminates this assumption. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Acked-by: Chris Li Reviewed-by: Ryan Roberts Reviewed-by: "Huang, Ying" --- mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index 78422d1c7381..2702d449880e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3856,7 +3856,7 @@ static inline bool should_try_to_free_swap(struct folio *folio, * reference only in case it's likely that we'll be the exlusive user. */ return (fault_flags & FAULT_FLAG_WRITE) && !folio_test_ksm(folio) && - folio_ref_count(folio) == 2; + folio_ref_count(folio) == (1 + folio_nr_pages(folio)); } static vm_fault_t pte_marker_clear(struct vm_fault *vmf) From patchwork Tue Apr 9 08:26:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13621995 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99E30CD1284 for ; Tue, 9 Apr 2024 08:27:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EDD976B009A; Tue, 9 Apr 2024 04:27:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E8E356B009B; Tue, 9 Apr 2024 04:27:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D2F326B009C; Tue, 9 Apr 2024 04:27:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B09266B009A for ; Tue, 9 Apr 2024 04:27:13 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 78ADEC02C7 for ; Tue, 9 Apr 2024 08:27:13 +0000 (UTC) X-FDA: 81989313546.13.CF3D9B0 Received: from mail-pj1-f53.google.com (mail-pj1-f53.google.com [209.85.216.53]) by imf13.hostedemail.com (Postfix) with ESMTP id 9E60C20005 for ; Tue, 9 Apr 2024 08:27:11 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=WG9nMih4; spf=pass (imf13.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712651231; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9ZXg/AhadeS891XoBBBMkIgTpnMXwbdxyyz0Yc4z5x8=; b=D8ukdxalbrgUONxZFQixYUGOLQMv1K7QVJ+QoHXdexLmlsQK6GWBH4XCrc969i/VAJUH+j AaFnHBX076RiJAq3iUKtHFS6gY+6P6uFOp25B+gHH5wVrnaEg6sAx6smSp+tnEy6YWj9HW Q3H9E4jkeilQFW97lp44cxFjngbCJgQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712651231; a=rsa-sha256; cv=none; b=pgmxHqMID/gpJYsZjDK7a1NBjW5DaG/qfuLolBXrMZctgYW/Cry5ewEvPotm1usTRmCIXE MfAreetK3uVRo5VwR3J3VmXtS2ZMvlJdxhWyO90aDWxWqhVXhb8uGnf7SOWMKxgv7C3TXa aOMWCyQRFm53cY2tAwZcYDBGyAUwz8Y= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=WG9nMih4; spf=pass (imf13.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.216.53 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pj1-f53.google.com with SMTP id 98e67ed59e1d1-2a52c544077so1216801a91.1 for ; Tue, 09 Apr 2024 01:27:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712651230; x=1713256030; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9ZXg/AhadeS891XoBBBMkIgTpnMXwbdxyyz0Yc4z5x8=; b=WG9nMih4kxZ2dM6X98GQXSAuurlIhbkCtAGL7qcrKWjpCRd++aQnRzPlHW4rTbOGAt paM2tWD50L2ytQN+dhAvGdqkHx5/VXIo6xWunG0Tq9We5NaefaAcqGm57LYRVavm/SD1 +E/OHyAZuA3icBOD7MBMPuH/YGR/WFz3QYAzzgAfIMz8BJqlVRx/ruuGa7CRfsnxkueU PccstpM6RSzv8rIfsHefKd2Mgrcb0aCqC9H6jbaNMGjkED2LqgwTTNTlsbsLS4i4MFIP Z0fU3NY+tuWlhPDPz/rlI8mQ+jchtLmjEcurPJbXZyhrNp5CdBLkZL8Mv/bjAwd2uNCI EmzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712651230; x=1713256030; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9ZXg/AhadeS891XoBBBMkIgTpnMXwbdxyyz0Yc4z5x8=; b=YGxW/mnNw5hE9Zh7twyakldsOzQsNHgImvk4wU37eSKLJCKhrv3RxqAGG9nE5tG92V n7Yu/sbkhR1cpni9lJxtLgnOGaljzWdRyx+dZPoEOZLma48ycXg7jk2WUqdz6epIh1hk 1Ki3pQWRK5oz4n2z3uLZRMlcdcSY9x2vwJUF5f+cwGUrEbTAldNmQnkOOL9BN3pc41s+ eZNVAx4eFTMlx1/vXXE/GKQCnJsJdVnm0r17hCFyAgnQcL3waecXl+FB3s/z+aZHcPgA /R7Py/HQ4uPiYHbhMQI0ADr+sLC2GZCzU+0bBiZxaqfTjLSPiFpYyC/Aw6Hks421H0j5 xFjw== X-Forwarded-Encrypted: i=1; AJvYcCXyXEsZEHG1hD6rulw/QKOjT2xIrRk2wFeHjrLANynD6c3YFlaeNMv5hzwjuPZy0C0udoL2N+hrx4lfyWgNYjqB0wM= X-Gm-Message-State: AOJu0YxWELrjjNpezhSmkh8pOZdUuK1xDBvefU0yGwjJiMewX3MnCAqo 35y4k30j8utvfMEB4FXKvP5GUaEwZGe+L9j5in3ymCXBtiUFRS3U X-Google-Smtp-Source: AGHT+IFg5AbIygO1G/4Sf/xhmF54Nd1zJbM7cGgBTgS8FOC9fn0UWuAxj2wjNfCbUfjBKgj68N+xMw== X-Received: by 2002:a17:90b:3587:b0:2a4:e962:b62c with SMTP id mm7-20020a17090b358700b002a4e962b62cmr4228631pjb.49.1712651230311; Tue, 09 Apr 2024 01:27:10 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id r16-20020a632050000000b005f05c88c149sm7594238pgm.71.2024.04.09.01.27.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Apr 2024 01:27:09 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 3/5] mm: swap_pte_batch: add an output argument to reture if all swap entries are exclusive Date: Tue, 9 Apr 2024 20:26:29 +1200 Message-Id: <20240409082631.187483-4-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240409082631.187483-1-21cnbao@gmail.com> References: <20240409082631.187483-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Stat-Signature: hqtnxsup38mk9z38i7sje58ocrboi9qg X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 9E60C20005 X-Rspam-User: X-HE-Tag: 1712651231-856992 X-HE-Meta: U2FsdGVkX1/XBwazRswzB73dN5RD3HVxL2WCR40yZYGtvKo4FmjleGqtvbni1gpcRHGSHPA1T1DlyOH1q5JZKf8VrYRzTXwSxsT8dw6cPujWDT7iqiT6KpmZ13HHGODL7VcYlcdJNd+eRFINCteb+XgTyWQMIMi4H3ZB9EkAdGLQ/dp1MGngTbh8oxgxaip2QyrjTq8n4RalkSSwLAVYShPvL4yYemAYYCFqdMTq7j1rpr+qQMW4vHWQ3jvSewfVK5Uj+oE5MBKzic3EC/eoS1cUbcxbIvE22+T8hTkV8+W5o9NrfS6O8W5QEhrcyWg8WJIGRH0w0k3Qvj2wbU+z4NxCCDcgUqZu664mVfYOJoT9Y6XmjuQKwGp6IJmA3cRZlWjXU6aCAXIjYz3z3dSlWnAHSDXdfMu7sOHK7+Cp44IS1vyecsj5I1i27ez3qJE4Q4ElxS45xLcR8DtkoZE5No3tDUkHUZDZvuFY9Z0xRJ/9H/AuX3KxzAQmlv9kmf7pr+5cMiz0TabZBrPpUvvoOaIBxTf1S2at8nszcY/WwugEPOYREZOwmysibCmQX6sB2D42GgrKsfXSfwxLAjujLEKYsArjaapmOg4WXoXhSTNa04/YRYOY08DO5Bwo7AgJGtK09JT8yDo32ICIMBAVRuu3hC2L5Y/MejslDPBVFkkSJ+2FcRrHGB5lzaUvBh6FJ8hLu6xB57SJYTD3w8ymZYBZ6Mr8GvJ/OXgsvL+DbTCs3XHooJpSNvJvuKo3ydNL720FCLbDxW5C2quACbVvvfqKRFGpTaYfX8R7voqxfRydpaKbBOYMu6r9yZABWVzPiTTssbaofPaJBjboa9pFugJx+YKmJqm5OtXIaQTaT6jpc97eUy79r24sKhq3Rsoy68OoJOU9Sz6gsTtQ8eKvDysg8ldZQ1uSG26+vtZrcA2KUXTD2SzSCk2RT1izehtTqSN2JyV8K317157q+Fl pkUTzpVl sdT0H9ccHW8eUQjz7zCjLOqEoP13pWiOSfkOngCt3Sj/VH473UZwN9azrKV506bxlI0uwIQ6acBIyDrIA0b8L+OFnuZ77BQIvJNNKvHYJAsqUgfdRs8DODlbfGO85A+8YXbQ+utPFgES/Fa8JPeunwl6lIdmqOMxAIGaOnaGiE91GtfA0/X1kdtA+t6HgMWUkxWCgjQao9+ziuCHVVDf0J9O85NmiZu8x2oWYoyy5TI2UY/wQaGlerpF98oEmLRrFxFDN8TDzajO4E+upMJR+dUm1h1ZZ4aM9QToTMhy6iPvsypOpmBfgmH4B5ZZqTb3ionri1R230cAlPEAMxcgdkgUOuLX5Z07M1VUnCbL1Va6zOZ3z9PomRXSzCE05VgSERxpSE0b2TWq6yMlfAzyiWUWX+5Zb2VkyQ01HTtO4a/GDI0G1XSl7ntnJX4z42hMJV5kd8H8/BdUlySYQ741Ww7tq16kB2cF1S/zN9VRVgYOzQQeIASf843rtEfg4g9pGTxA0GEJdgqtnyWoE2wgH6+p1RF4ZPJCw3pNOrud8vxWeVTmmjLR3/bBcYxf6Dr3OyYs7 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song Add a boolean argument named any_shared. If any of the swap entries are non-exclusive, set any_shared to true. The function do_swap_page() can then utilize this information to determine whether the entire large folio can be reused. Signed-off-by: Barry Song --- mm/internal.h | 9 ++++++++- mm/madvise.c | 2 +- mm/memory.c | 2 +- 3 files changed, 10 insertions(+), 3 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 9d3250b4a08a..cae39c372bfc 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -238,7 +238,8 @@ static inline pte_t pte_next_swp_offset(pte_t pte) * * Return: the number of table entries in the batch. */ -static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) +static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte, + bool *any_shared) { pte_t expected_pte = pte_next_swp_offset(pte); const pte_t *end_ptep = start_ptep + max_nr; @@ -248,12 +249,18 @@ static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte) VM_WARN_ON(!is_swap_pte(pte)); VM_WARN_ON(non_swap_entry(pte_to_swp_entry(pte))); + if (any_shared) + *any_shared |= !pte_swp_exclusive(pte); + while (ptep < end_ptep) { pte = ptep_get(ptep); if (!pte_same(pte, expected_pte)) break; + if (any_shared) + *any_shared |= !pte_swp_exclusive(pte); + expected_pte = pte_next_swp_offset(expected_pte); ptep++; } diff --git a/mm/madvise.c b/mm/madvise.c index f59169888b8e..d34ca6983227 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -671,7 +671,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, entry = pte_to_swp_entry(ptent); if (!non_swap_entry(entry)) { max_nr = (end - addr) / PAGE_SIZE; - nr = swap_pte_batch(pte, max_nr, ptent); + nr = swap_pte_batch(pte, max_nr, ptent, NULL); nr_swap -= nr; free_swap_and_cache_nr(entry, nr); clear_not_present_full_ptes(mm, addr, pte, nr, tlb->fullmm); diff --git a/mm/memory.c b/mm/memory.c index 2702d449880e..c4a52e8d740a 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1638,7 +1638,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, folio_put(folio); } else if (!non_swap_entry(entry)) { max_nr = (end - addr) / PAGE_SIZE; - nr = swap_pte_batch(pte, max_nr, ptent); + nr = swap_pte_batch(pte, max_nr, ptent, NULL); /* Genuine swap entries, hence a private anon pages */ if (!should_zap_cows(details)) continue; From patchwork Tue Apr 9 08:26:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13621996 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1FEF9CD1284 for ; Tue, 9 Apr 2024 08:27:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AE8346B0082; Tue, 9 Apr 2024 04:27:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A96A56B009C; Tue, 9 Apr 2024 04:27:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9373D6B009D; Tue, 9 Apr 2024 04:27:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 763CC6B0082 for ; Tue, 9 Apr 2024 04:27:20 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0D0E31602F6 for ; Tue, 9 Apr 2024 08:27:20 +0000 (UTC) X-FDA: 81989313840.13.0E4FA90 Received: from mail-oo1-f52.google.com (mail-oo1-f52.google.com [209.85.161.52]) by imf18.hostedemail.com (Postfix) with ESMTP id 5F7191C0011 for ; Tue, 9 Apr 2024 08:27:18 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=dC2jeEj6; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf18.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.161.52 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712651238; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pRaJtzo5cg/VrfJljXw5TEaEgpb4vopERkweBTeE6OA=; b=KxJFmjBhmYw+/E1Wm0OrTl9ZnFxrmtZv13iroaXEa5023Jfi0jHOBx+9tMDTYnHZxG6xyk JV0UQND3L7IjLEs92+u5HKufeUpObkFc7lPDLbgfOkkWdsj0IoIer/LT4cSEie8o4ujDW3 wMZmvn5vtdzySDouy09dAe61BMIy+tM= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=dC2jeEj6; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf18.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.161.52 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712651238; a=rsa-sha256; cv=none; b=gJeO69qYwG7CkiRRucnUHiYh3dhOE5emNsKPCjpHRa1sT/Bn37Ge9oHdJza7ojEdzvu4bA ZWv6loHH6DxnLxwpnDV51Y353wZxjv6voB/VuMNEV1o82i8kYyoLM6B0Tlpntp3mZR6pP1 ODOp7/N6IADayoiJXvEVww5295uIpho= Received: by mail-oo1-f52.google.com with SMTP id 006d021491bc7-5aa1b7a37b5so1725093eaf.2 for ; Tue, 09 Apr 2024 01:27:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712651237; x=1713256037; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pRaJtzo5cg/VrfJljXw5TEaEgpb4vopERkweBTeE6OA=; b=dC2jeEj6X12s/S26d9UZEFdjciYeT2anEKQ8bQAGwXwFxv1tdYgGJ3zaddvutpH3k6 wdVd+hke2a+su6z9TM2wbB1wnnIdxkPaci5sDk9SH/nW98X0/ivnzpMVuM7RpBwUKKLR bahcZvgkdnJS16yQ9LBf56xq//CWXG+EDo2xfInts1FdOUg1MR/pRLuUtgqKQtcW47Uz V0vx6U03VtYAJf5H8BySemLza6VtgaSRiKWhSyNzgyZ4tx7CvVCXd4XPAB9H16O5r2+V yhrFTTMtdgrJKHUcAhh9RhcrL5mepkb+l+AauNDl2KuQKY2uX2aq3WvP+47aARxPqSKc j6GQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712651237; x=1713256037; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=pRaJtzo5cg/VrfJljXw5TEaEgpb4vopERkweBTeE6OA=; b=TqpWCwqctCaD4buS6fUcYudMV2cXMtbDxwRRY8bmpK6eNC06k9SmwrLQZdcYLy9nLP 5+m+F7b3S+SRzNbZRRhPS91OMSB2HwLJQfNsgiWCdsy7bYdfGqmfRyxAGHSi3eV4nkLG LfUdxcyz4m3qjRGoDJJx0NwnVAeHCcxmUJw72TUlzCHVYI3f5fhm1c3nHuDCov2WWROt 8heGP7V2a6Y/jMKVB9QvKAtwSdp1D4AZgZK3WcXGFpNmHN/Ysbfooh1KU+LoIM0Sdl7t yeJ8gEu3djEoCufqL/XDEf+S/tGIwGe9LbfA51I2NXmpIU0TYlPcWnSCtEqi6v+/piMl Av5A== X-Forwarded-Encrypted: i=1; AJvYcCX0+XGwwFcKP66zaAAIUo56corWk4bFzYQY95ugsrIA9cUQ+cM0Pi6HD94V16a594rvxDfBWK1TWDyeSjZHp2eyhcQ= X-Gm-Message-State: AOJu0YxO6oj+ubbb9OVOvkYZQ1LNhjugCVlNJGv45RTIdltUVukIovY6 bCDWmfUdHP+oSM4IdYK0ckIye48jHgZYpy1twbkH8YVO4FDk7O74 X-Google-Smtp-Source: AGHT+IFvN5ohj5Cxouk2/0fAxnzfDvT6AqBlF178427aMryNUnuyIkM8ZO3KggJQq1sFBYMpf6zx2g== X-Received: by 2002:a05:6358:d59b:b0:186:45c:d6de with SMTP id ms27-20020a056358d59b00b00186045cd6demr6322611rwb.26.1712651237158; Tue, 09 Apr 2024 01:27:17 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id r16-20020a632050000000b005f05c88c149sm7594238pgm.71.2024.04.09.01.27.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Apr 2024 01:27:16 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 4/5] mm: swap: entirely map large folios found in swapcache Date: Tue, 9 Apr 2024 20:26:30 +1200 Message-Id: <20240409082631.187483-5-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240409082631.187483-1-21cnbao@gmail.com> References: <20240409082631.187483-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 5F7191C0011 X-Stat-Signature: 3d33umup4mzgwryq9d55onctscjq9qie X-Rspam-User: X-HE-Tag: 1712651238-671154 X-HE-Meta: U2FsdGVkX1+7JDQBwQkhOMkRGxCDQE3cVoopAtmmaZpjOAiWy6ZbiO+ukQCok1U679q2hdxTXARMJvHjRJJlrhPvvDMRcU6Ybc0kc+KROFDyhSHuLkJndORid8a3iIdCMGVGnYdE0MjTH/dFHf4QyzcnmDW8r9UxnD7rc1Db+MADaxnypbXKOrZs5bnU/n0liF2TGI7Lmo8q9nhFyr0MtlCM9X/85DiqHD/S3TEDQAJcMPDbJkaay1GQst6YMYN211TSFtM7JAFtHxYouFm7iMZfo+ctlvpB9lWAL3KnpOMt0ZFfrfIqzMghHDtytE/0HL4n2u2bC4DWW/RGtt7gGcLenHgepiFWuFkglUdfOMK+Y53JwqUi3ULVK4x1Ds1BOI3HCRUSiKay7eYpv+1y2mGgc8lr6RWJCJ7+eBeOLMdAmXoT9Vou9VdBVsTkYCYwISGzdNaBNJWHX4HUv4tV4q+7sK9gRj97Xw6fDUyfbNVV8IBwRb8mhlQqKp7tGwdX10sq0GsgZEnbyvpv9zoXdnngbmSSsiO1afpcvsofAzKWmOyGz4Vfrm1H7BiA1nCcOQfvZV8knetC7hN+WfpKX0ly45h+HL1VxcKx3bs0lAkwLNp+Pwu5GVgMN8W9s/jaINAcYVc4CAqO6dTzMMiiWRc1J2KQTDbYMtmCQTDJqF9stpKFS752XBWGvjENzOAVWFKbu18MpUYLQR8YMdaP6GH0VZVOWdIQgVu3KrNWdFmeCgvALpUvc7mmYWkTeqmmuiiTLIJ4EA0qxtjAJREX60RZ8wEwYsPb2F4uBl2kQheH7uB/87f1KNwgd6tzxUyrh6nwY2aJZ9RSZVeJI0C4xHWFTOREH8pLFtSNnlqVBC+HKQwXhzIPH2V+Z7boJThIPfySZEa9c4kmWLj4AX8XHa/dww7L4hFJrjFrvRXWZbOXo1CfeocTTttYL2HnWLh5fC/gahCKdtTRWcyxhbY 6P9AVvpS TnLTjHEQ0MEhIbFV0ViE3f7Ix7nm9V2zFz4b6wqOGiV9M2LBk0UPrsy4EqhMD7/NJt7ENR19ZANK22f8h84fJUvay5sUofeQiJ3snxca6J/C35T0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han When a large folio is found in the swapcache, the current implementation requires calling do_swap_page() nr_pages times, resulting in nr_pages page faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song --- mm/memory.c | 64 +++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 52 insertions(+), 12 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index c4a52e8d740a..9818dc1893c8 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3947,6 +3947,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) pte_t pte; vm_fault_t ret = 0; void *shadow = NULL; + int nr_pages = 1; + unsigned long start_address = vmf->address; + pte_t *start_pte = vmf->pte; + bool any_swap_shared = false; if (!pte_unmap_same(vmf)) goto out; @@ -4137,6 +4141,35 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + + /* We hit large folios in swapcache */ + if (start_pte && folio_test_large(folio) && folio_test_swapcache(folio)) { + int nr = folio_nr_pages(folio); + int idx = folio_page_idx(folio, page); + unsigned long folio_start = vmf->address - idx * PAGE_SIZE; + unsigned long folio_end = folio_start + nr * PAGE_SIZE; + pte_t *folio_ptep; + pte_t folio_pte; + + if (unlikely(folio_start < max(vmf->address & PMD_MASK, vma->vm_start))) + goto check_pte; + if (unlikely(folio_end > pmd_addr_end(vmf->address, vma->vm_end))) + goto check_pte; + + folio_ptep = vmf->pte - idx; + folio_pte = ptep_get(folio_ptep); + if (!is_swap_pte(folio_pte) || non_swap_entry(pte_to_swp_entry(folio_pte)) || + swap_pte_batch(folio_ptep, nr, folio_pte, &any_swap_shared) != nr) + goto check_pte; + + start_address = folio_start; + start_pte = folio_ptep; + nr_pages = nr; + entry = folio->swap; + page = &folio->page; + } + +check_pte: if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte))) goto out_nomap; @@ -4190,6 +4223,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */ exclusive = false; } + + /* Reuse the whole large folio iff all entries are exclusive */ + if (nr_pages > 1 && any_swap_shared) + exclusive = false; } /* @@ -4204,12 +4241,14 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * We're already holding a reference on the page but haven't mapped it * yet. */ - swap_free(entry); + swap_free_nr(entry, nr_pages); if (should_try_to_free_swap(folio, vma, vmf->flags)) folio_free_swap(folio); - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - dec_mm_counter(vma->vm_mm, MM_SWAPENTS); + folio_ref_add(folio, nr_pages - 1); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); + add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages); + pte = mk_pte(page, vma->vm_page_prot); /* @@ -4219,33 +4258,34 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * exclusivity. */ if (!folio_test_ksm(folio) && - (exclusive || folio_ref_count(folio) == 1)) { + (exclusive || (folio_ref_count(folio) == nr_pages && + folio_nr_pages(folio) == nr_pages))) { if (vmf->flags & FAULT_FLAG_WRITE) { pte = maybe_mkwrite(pte_mkdirty(pte), vma); vmf->flags &= ~FAULT_FLAG_WRITE; } rmap_flags |= RMAP_EXCLUSIVE; } - flush_icache_page(vma, page); + flush_icache_pages(vma, page, nr_pages); if (pte_swp_soft_dirty(vmf->orig_pte)) pte = pte_mksoft_dirty(pte); if (pte_swp_uffd_wp(vmf->orig_pte)) pte = pte_mkuffd_wp(pte); - vmf->orig_pte = pte; /* ksm created a completely new copy */ if (unlikely(folio != swapcache && swapcache)) { - folio_add_new_anon_rmap(folio, vma, vmf->address); + folio_add_new_anon_rmap(folio, vma, start_address); folio_add_lru_vma(folio, vma); } else { - folio_add_anon_rmap_pte(folio, page, vma, vmf->address, - rmap_flags); + folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, start_address, + rmap_flags); } VM_BUG_ON(!folio_test_anon(folio) || (pte_write(pte) && !PageAnonExclusive(page))); - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); - arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); + set_ptes(vma->vm_mm, start_address, start_pte, pte, nr_pages); + vmf->orig_pte = ptep_get(vmf->pte); + arch_do_swap_page(vma->vm_mm, vma, start_address, pte, pte); folio_unlock(folio); if (folio != swapcache && swapcache) { @@ -4269,7 +4309,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } /* No need to invalidate - it was non-present before */ - update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); + update_mmu_cache_range(vmf, vma, start_address, start_pte, nr_pages); unlock: if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl); From patchwork Tue Apr 9 08:26:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13621997 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BE13C67861 for ; Tue, 9 Apr 2024 08:27:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 346246B007B; Tue, 9 Apr 2024 04:27:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 300436B009E; Tue, 9 Apr 2024 04:27:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1BD436B009F; Tue, 9 Apr 2024 04:27:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id ED1EB6B007B for ; Tue, 9 Apr 2024 04:27:26 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id AEE831C049B for ; Tue, 9 Apr 2024 08:27:26 +0000 (UTC) X-FDA: 81989314092.07.309F07E Received: from mail-oo1-f47.google.com (mail-oo1-f47.google.com [209.85.161.47]) by imf04.hostedemail.com (Postfix) with ESMTP id 0D6EF4000E for ; Tue, 9 Apr 2024 08:27:24 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=O6coziRY; spf=pass (imf04.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.161.47 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712651245; a=rsa-sha256; cv=none; b=ob2nkeBzl3lp+2fGIVCHWSilXALUIIXfr31gNWfPMIv89k6mFLr75BTLA4L3PbKvcvhKnj knF0ewYVqIfVJU5uXQIgpiimJ4p8E+4Ck3by3HXIibACXycS6TPgvRJgwcf95QswneQMwZ 9E+cQ4UQJd+w7kibaO4M+y8Rh/GFMHI= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=O6coziRY; spf=pass (imf04.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.161.47 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712651245; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VvJHnREnFZ1cH5hEIu5QcKzQyYhaFholreUs1EUriRY=; b=gAKlrRv3uvDJ983CDgZ0dMuPyXaahOqMKKa3DPNszpgNJREfiXiFzv6yDmehxavF3CQwZ/ xX6lEp4LYxwMuIt5OE52SosVZRgnkfpA+LgMOqdi+goE0JOS892ovUl1Bb5K+LXgK7HUPw Sg/zVKUV32IczMJvzRFcw0mbpYcJdx8= Received: by mail-oo1-f47.google.com with SMTP id 006d021491bc7-5aa20adda1dso1979600eaf.1 for ; Tue, 09 Apr 2024 01:27:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712651244; x=1713256044; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=VvJHnREnFZ1cH5hEIu5QcKzQyYhaFholreUs1EUriRY=; b=O6coziRYUBf3HXvH1smalQLi5L6Si7xi2odywHmhKFC5dVT17YnY4ilyk3wDz3CC2P n650SHrrg41zGW4UEEDg0leap13WOGttB3aIMOvJm/rvr/2fbmuUU0LtyaMXaHf2/M+3 Kj28S2GZBC2e9R6yeQJ/uIODzgm0yKllSYFeDvmcDAT8XWJ3VyH/cxt4F6lOrT3hltNU 6TD+8t2uqyKxtqWHHZn069IICEoNO+K1FJBmZGk2hnBuggnto0+8vZbuzRAxzIZW2JUx tHmIXogOb6kchoYt1bXg1qYTCoRbo4PwN8ivtQ6uFUsZQhKUrUJzzBSUlR4BtmUHcTOQ Sxtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712651244; x=1713256044; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=VvJHnREnFZ1cH5hEIu5QcKzQyYhaFholreUs1EUriRY=; b=GKNoXqvsKvOnYhMvNEWKGD05nWZLT+zer2T63RcKXK3ZEuq0htgp6Y/iywFY4r5rA9 zSj5SY+jWPVn8cU63xDCsozyLHwLl8ISjuAXVEmJ2bgH9+HzLh33rmCjQhfaN4IDwmMx 0Fyub6R3G9CRVYQKAPwHhzDbPB5e43SjqK220kHvzDcB/RPHgbvdqWWOPNmzGUQuOr1P 3UbfUyZj5T8ZFPX/7dq5PmxyP0DbydCDeFifRn91ArcVpp7FViVrPrBf6dyhJyU8p1LN Z37NC1Y3FE9AtzNDTm0ZkJSOJeWvVLQTOCJSkxVj0sFWLf0/r4uXxEbkeIebedFM7JQT 9dcA== X-Forwarded-Encrypted: i=1; AJvYcCVwUgGrwAPV6/La7KlrFmKnbaRfIrZwdNIMle+Jr4/LPT4E7gqI0RCwv6fum8fzIH6C/Wjo70qbn/Scf4bNNYAuhjc= X-Gm-Message-State: AOJu0YxB6622gwX1osj1ApNKIcnUqAf/0IFATBOWHm0YXQWkaYspDC4m eJMMcmZeLJCTYIqdINE+j3fl1KCk7AGueQMerHpTpWdygxa6uLoJ X-Google-Smtp-Source: AGHT+IFBs0lnKgGergw9nSynKWYDcWV9NAHxsayy25kkPHq1gK1xpsfE+xyR/l1fPI85a7fcICPnHg== X-Received: by 2002:a05:6358:d39f:b0:186:102b:777 with SMTP id mp31-20020a056358d39f00b00186102b0777mr9881254rwb.10.1712651243980; Tue, 09 Apr 2024 01:27:23 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id r16-20020a632050000000b005f05c88c149sm7594238pgm.71.2024.04.09.01.27.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Apr 2024 01:27:23 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 5/5] mm: add per-order mTHP swpin_refault counter Date: Tue, 9 Apr 2024 20:26:31 +1200 Message-Id: <20240409082631.187483-6-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240409082631.187483-1-21cnbao@gmail.com> References: <20240409082631.187483-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 0D6EF4000E X-Stat-Signature: iwt6ae9wn3yaqxu9bucsz7onpqdf3k3m X-Rspam-User: X-HE-Tag: 1712651244-702132 X-HE-Meta: U2FsdGVkX19NkgkET4PsPsy11SPQXfVjBpZiFVmaarz7m/tABQk6MPM1Z7C/LrplsGya62sT7e8QgqYwOj8kt0JPa8/kx5fAkTWlGidjzFZ80XOiaBX1H59xe24FPBGaT+fgUXQKDATPXGDO40gpebqOIPTXan+P7QF1FRN87gAy6GhL9vL9q/fn0lUMqpwUuGEXcJ0PMCrbVz6FScuBqD/ANy2J9WqUgTKHnqW7YSmRKW6Fb9pPM5hwSvlaqoDj9dkKW68xnpqzc008ssX9ir0yBXiwrGMTOMt8SRW0qtFfHd+FhBloAP8Dd42XHFgvVTlsKtQ2CfrM4Imj+KE6HeHCStkhggNUo5gNcvjod93cpiVcM9DLliCk94SSrDmgzXwe4n8KFMtS9jK7QIM2sIv8D5OgVKjUXVLENqlIcVG6jg4n/Tzu/Lq+QpBQUFynuROMXUgDiKdwVaKJOoEq2+Lej5YZRARm/aHsfNhOV2Rt6FjRNB7bYq5OdnaSv2PSIazLTeqLMVCXGCxgHa1o/BrLnEM8lnbIzCq4W7opNI1ykN6D8Mgk7q7TCF3T3IvcuYK8h6nBi8RRKEoSWSIFySU2yIkZKcO7BaT+9aYyiDpfKtaYFjYIitGfaZE/xMZ0vK52Gg19f2o0lQvbjDl8IH3gYl5HX1tRGqKyS0q6Y+i8TN9E0+KRDr/mAhSWM8mMf5FwgMwC7tXSk/JrGqniIa9k+jR5f9AZPH5BqgGCNrZ4aDNBMtt67DQiFR1FVQ2pK0e5+DrE5YzZ2tcEtIttKKMuR6gj2wRkg1XG2MHelMHrOIDeMdE5c0X0pko6IovH42q3If8kshezG20vwxmj7q6ziZ3yR3DpxB1+b3RY/s/Zp3Km79zEow8bgrBD6oA9+1aI9kEb+d/oTVG/DThG7P29b2fXcJbgI3dIsfnK7iX+eJGBp2tnvfw+lkDB3xEmbFkOL9GLTRIhH/unVaJ U4ghppIc I4UCpZaug+0uUGaLHOjJX/ZOF8fiR93DmjdhRTwmh6Heai7H9pHdFbMuMY3kGNlefUvjvYz4B97l+R95BtnHeSyk+SAmhcbb4pVdqkdCRrXO5UdHnppxOFP70bUgeKPLsez7xJPBhgv/LFgmo1jidVd7ZCuXDre8SJUxU X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song Currently, we are handling the scenario where we've hit a large folio in the swapcache, and the reclaiming process for this large folio is still ongoing. Signed-off-by: Barry Song Signed-off-by: Barry Song Tested-by: SeongJae Park --- include/linux/huge_mm.h | 1 + mm/huge_memory.c | 2 ++ mm/memory.c | 1 + 3 files changed, 4 insertions(+) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index c8256af83e33..b67294d5814f 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -269,6 +269,7 @@ enum mthp_stat_item { MTHP_STAT_ANON_ALLOC_FALLBACK, MTHP_STAT_ANON_SWPOUT, MTHP_STAT_ANON_SWPOUT_FALLBACK, + MTHP_STAT_ANON_SWPIN_REFAULT, __MTHP_STAT_COUNT }; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d8d2ed80b0bf..fb95345b0bde 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -556,12 +556,14 @@ DEFINE_MTHP_STAT_ATTR(anon_alloc, MTHP_STAT_ANON_ALLOC); DEFINE_MTHP_STAT_ATTR(anon_alloc_fallback, MTHP_STAT_ANON_ALLOC_FALLBACK); DEFINE_MTHP_STAT_ATTR(anon_swpout, MTHP_STAT_ANON_SWPOUT); DEFINE_MTHP_STAT_ATTR(anon_swpout_fallback, MTHP_STAT_ANON_SWPOUT_FALLBACK); +DEFINE_MTHP_STAT_ATTR(anon_swpin_refault, MTHP_STAT_ANON_SWPIN_REFAULT); static struct attribute *stats_attrs[] = { &anon_alloc_attr.attr, &anon_alloc_fallback_attr.attr, &anon_swpout_attr.attr, &anon_swpout_fallback_attr.attr, + &anon_swpin_refault_attr.attr, NULL, }; diff --git a/mm/memory.c b/mm/memory.c index 9818dc1893c8..acc023795a4d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4167,6 +4167,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) nr_pages = nr; entry = folio->swap; page = &folio->page; + count_mthp_stat(folio_order(folio), MTHP_STAT_ANON_SWPIN_REFAULT); } check_pte: