From patchwork Tue Apr 9 08:26:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13621993 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52948C67861 for ; Tue, 9 Apr 2024 08:27:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D7EE26B0095; Tue, 9 Apr 2024 04:26:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D2F316B0096; Tue, 9 Apr 2024 04:26:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF9DC6B0098; Tue, 9 Apr 2024 04:26:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 9F94B6B0095 for ; Tue, 9 Apr 2024 04:26:59 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 5C7A91A02D1 for ; Tue, 9 Apr 2024 08:26:59 +0000 (UTC) X-FDA: 81989312958.20.D3C02CA Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by imf18.hostedemail.com (Postfix) with ESMTP id 88C801C000A for ; Tue, 9 Apr 2024 08:26:57 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=P9ORVcsi; spf=pass (imf18.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1712651217; a=rsa-sha256; cv=none; b=WtuOc03Ze7YN7QNQaQv3Dwg+5joL60esj/8o/fMdjH3V0Z0wxWfHcTgSHUxxoRRxgIay4l 7TuZutOpDuo+bA8ilo4Bf/zmFvSUUn8WLnle3vWLkkjhg0U4K8f0x8rBkE9r4qp/XUxiyL WkB4wAJ9bEHqti3oWqiG0ECVh0fQrDY= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=P9ORVcsi; spf=pass (imf18.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.179 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1712651217; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ei1Crh3xKYkUssDLy2Y3ak8IjHOYPwPhmj2M8HC/g7Q=; b=Fgwc3HZum97w8vE+n3eqf/8ZJx0JPb/UOpQmICr99z5xYzFSf35smqx2LLghCzYNBrCigr s0J4Ur8SAyixcPMVtti+cRzv1W4UbYHSoazdwvIQG2g/qC1W0i1Dc0PC/Yi7YOUAXvns4W 6KIJbRDIewcxF0p2ZBJohGLV9fN2H40= Received: by mail-pf1-f179.google.com with SMTP id d2e1a72fcca58-6ecfeefe94cso2614595b3a.0 for ; Tue, 09 Apr 2024 01:26:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1712651216; x=1713256016; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ei1Crh3xKYkUssDLy2Y3ak8IjHOYPwPhmj2M8HC/g7Q=; b=P9ORVcsiLL32THHaYzbIHOK2l+jQDjlDtxWAcvM/LT3OYt00o8gIKejUW4JdiWQdVZ 1qu6HAqni+EkATX24F9Ov0IrjpLiV1AlcrebrEs96JH8dFTf/GzqCIr756Gm6UU2jntn 8DfZREeMGXGzN1kEUqjQzEY7G0TrmPCtzu7ZEmonhjJFbuGxJZpAOLD1KM/uY6WNkMHX u5g9N9Stvz0tZb/S8J/WNeT/M0gUvTE9omeVgEl00nmwCDWU/y+JDWdPas9QE3RtCeVn r63vayyGqCdIIdWRFFGXMAYNcO1MIlvJYG64gQrtzrJ4KPgPeTR1vqKPPj95dV8V856b 6LBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1712651216; x=1713256016; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ei1Crh3xKYkUssDLy2Y3ak8IjHOYPwPhmj2M8HC/g7Q=; b=SDmZNXBzG+zzhFk+FLwTe2L0VlXKaptUNqZbenWjmSIIhQ7n3+OPf0ZnmzZGM6BNh7 lMHU62VerlDxcWf7ha9SSOv47MH4z9MeVjOgti/vIAG4kvutr9ib3ji72NifzYlIRnGu 2aJwEHVGEVoESqscDZuogJBk7H1x3hQ5caw7PT/KdafbelbwR5Bacx2hNrqrhNRzBt2k 7NCzJq7Dw7uAdvx1y05HsPJvS0JEs3XbttNe5spIYVxW0XcwuNww8OFbF/VPu/cG4+y/ /r2gjql0XJM/yDTeBzTaAO2PXC4yZmFJsEn4QwnIy52jM2Iuz2eQMlknnY5TUV5TGqqf hrIw== X-Forwarded-Encrypted: i=1; AJvYcCXpm0NHtmLKhJBeI0jA3N1ukiFeoO08gRpqrP75zt5to+N0nfgr10PLcxuUSZSdxq7fpF9DtVfG67gFLg9PaG563ZM= X-Gm-Message-State: AOJu0Yz3n2AuFcR/qnSLPoD4mOjF/z/YmqOl0jPMUZsLDONdsldVQX0G +/IoUjCoCND8MM7W8JPxkuCy+QSfjt18WiXQ5LIb9mjl0WxY5OWc X-Google-Smtp-Source: AGHT+IHZGO/8Oq5G4AI8d7uAa0Xfd/UfJnN/mN1F1Zgkmt5jIyqoC0gKP++Z8vrGbui309CoYrBrbQ== X-Received: by 2002:a05:6a20:430c:b0:1a7:5413:1e6d with SMTP id h12-20020a056a20430c00b001a754131e6dmr3319733pzk.4.1712651216427; Tue, 09 Apr 2024 01:26:56 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id r16-20020a632050000000b005f05c88c149sm7594238pgm.71.2024.04.09.01.26.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Apr 2024 01:26:56 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, linux-kernel@vger.kernel.org Subject: [PATCH v2 1/5] mm: swap: introduce swap_free_nr() for batched swap_free() Date: Tue, 9 Apr 2024 20:26:27 +1200 Message-Id: <20240409082631.187483-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240409082631.187483-1-21cnbao@gmail.com> References: <20240409082631.187483-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 88C801C000A X-Stat-Signature: zuncr9a7waehher8efjbjf7om6xdf9ak X-Rspam-User: X-HE-Tag: 1712651217-247917 X-HE-Meta: U2FsdGVkX1/icBn/OCkG8/xK+XNaPWuROp5mXRu7YYg2glnwuuZQadfb5RKW5BUGWNPo3K9GT+fMe7HCTzHhDAOvoE+NreJHYwvxVqYXNX0YUKE3XXwhE+oWnEbsIUcOYU6z7TO1cuieQ2IbBTx9ZnGbMBSWZ+MWj1h5QcJYcOklTGyQwJtGsj1Qhhb4FkA975yP33YlU9LStyHybBJV4epYpzGXERTFpU8/2Hzp9gMmnqO6pxuNqINWtljVns4NkIHSsKM+JL0trq6upkhiMLFoXMMeUQZCBoKMg9Fde2xCtvKI8oILtIej+fKLpNQfVT6nsKs189X2lKGWJVsZdi3d6M54pJWMDhC7+qv8XV9AOx6U4LWEwp9DdSJMnWzfbvMB7wYDaEPLq+Bx6czDcjmvlz2n4xmycoSwm/1IFOD0Tpd+6NFTjxX+jjZLxT1JTj3NMg9w8Dl9oU0u7eC2x1cI/p14KICMlI0B9g7DFkC2ySCT4R50STIGmms6qvFrQt8DLQ5THa/sKwNHfEEFWYlUFmPev6U2zW6iR19pUIO/964KNGWkucd7Kt+viaxbt9LUHnuLw8RIAJvPlBmQJ+zUz2dYCw4UAXJBFKZH/5J2BIVzhY+dNUw8+5Bae3ylfXhAu/vVVV2+/4VTCbJyReO9pm19WmUHCwc8JeMOKObD9Jt/eIRTDAoCb3FjarsQWs4eXXeV384Om/8TNfsGicc10okOr5nfvOMXVJB4uOqHOqOaWRHut76wE/vAHX994uw3pwCLQdfgkOHpmwj3WQm730/zILWK4zOspwlzI5INFHUiHRG05Iheu1UmTQQYjW4mcCIzkxK3ArwHWEdoHOWl0H3AF2KHF6fqDnzuyceksXVOlJ+IRUree3SJh7yxCFmUqBjdUPW2KAGJPGA59+ru9V0L/NHs+2W6ocoGyrlHjciqM0Ge44L8dgCKP4IDLdFeE8/B3I7h/10gF7J sqJgUaoU I0g9zRECxqWBu7Toa0A7scEDtbsv174rNyZNdwwWDXqKiRRpZdhMNpjCbO3r8fiMX0CSo4hrJ1y3dT+/kqBdPRjWhGFPmiENeL2JCw2VQrxTaO6xr2ahI8C13Afly7M0qz7jOTyrFZ63JVuPBatZhROWTbWJG10X4PksBt4Br0dHDyJAkIRyz/EQBqfsLd96cfffEJtz73hedp7/mt9F6u7kdCUcgNwl8S+Pn9404Z8S84wyOiMQqJ1uinn+7Fi+YC/65k13S4xu1yh4KdpmMRmvHRzvLL1WBuxox0ItrE1eaPdMTA8J8dJAGZvGt9E8rTOBb9Cvh14qy/G6HVr3hMe4whPeqNlHYClL6nrTYILSHrpd/kFUxLg+rg6cbQ+3WBdy+mJomT9E+vBIXd6rtPUohU61YG61e8ihkIfx5T3VpfkFjv21jbxUnCzScPgrga1qX/Am8CIiK+dNWMaNlfMdzoX4Z0K99KxfABK0hXbITUuwOG6SK8Bi8Y4BmXOeDINobOegtURPIlT5/HeS7+4MDU5zhWDm24rVWgoa6gIE1MtYTAW58HwFbOiFD64TN4phG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han While swapping in a large folio, we need to free swaps related to the whole folio. To avoid frequently acquiring and releasing swap locks, it is better to introduce an API for batched free. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Reviewed-by: Ryan Roberts --- include/linux/swap.h | 5 +++++ mm/swapfile.c | 51 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+) diff --git a/include/linux/swap.h b/include/linux/swap.h index 11c53692f65f..b7a107e983b8 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -483,6 +483,7 @@ extern void swap_shmem_alloc(swp_entry_t); extern int swap_duplicate(swp_entry_t); extern int swapcache_prepare(swp_entry_t); extern void swap_free(swp_entry_t); +extern void swap_free_nr(swp_entry_t entry, int nr_pages); extern void swapcache_free_entries(swp_entry_t *entries, int n); extern void free_swap_and_cache_nr(swp_entry_t entry, int nr); int swap_type_of(dev_t device, sector_t offset); @@ -564,6 +565,10 @@ static inline void swap_free(swp_entry_t swp) { } +void swap_free_nr(swp_entry_t entry, int nr_pages) +{ +} + static inline void put_swap_folio(struct folio *folio, swp_entry_t swp) { } diff --git a/mm/swapfile.c b/mm/swapfile.c index 28642c188c93..f4c65aeb088d 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1356,6 +1356,57 @@ void swap_free(swp_entry_t entry) __swap_entry_free(p, entry); } +/* + * Free up the maximum number of swap entries at once to limit the + * maximum kernel stack usage. + */ +#define SWAP_BATCH_NR (SWAPFILE_CLUSTER > 512 ? 512 : SWAPFILE_CLUSTER) + +/* + * Called after swapping in a large folio, batched free swap entries + * for this large folio, entry should be for the first subpage and + * its offset is aligned with nr_pages + */ +void swap_free_nr(swp_entry_t entry, int nr_pages) +{ + int i, j; + struct swap_cluster_info *ci; + struct swap_info_struct *p; + unsigned int type = swp_type(entry); + unsigned long offset = swp_offset(entry); + int batch_nr, remain_nr; + DECLARE_BITMAP(usage, SWAP_BATCH_NR) = { 0 }; + + /* all swap entries are within a cluster for mTHP */ + VM_BUG_ON(offset % SWAPFILE_CLUSTER + nr_pages > SWAPFILE_CLUSTER); + + if (nr_pages == 1) { + swap_free(entry); + return; + } + + remain_nr = nr_pages; + p = _swap_info_get(entry); + if (p) { + for (i = 0; i < nr_pages; i += batch_nr) { + batch_nr = min_t(int, SWAP_BATCH_NR, remain_nr); + + ci = lock_cluster_or_swap_info(p, offset); + for (j = 0; j < batch_nr; j++) { + if (__swap_entry_free_locked(p, offset + i * SWAP_BATCH_NR + j, 1)) + __bitmap_set(usage, j, 1); + } + unlock_cluster_or_swap_info(p, ci); + + for_each_clear_bit(j, usage, batch_nr) + free_swap_slot(swp_entry(type, offset + i * SWAP_BATCH_NR + j)); + + bitmap_clear(usage, 0, SWAP_BATCH_NR); + remain_nr -= batch_nr; + } + } +} + /* * Called after dropping swapcache to decrease refcnt to swap entries. */