From patchwork Tue Mar 18 15:06:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kemeng Shi X-Patchwork-Id: 14020350 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30204C282EC for ; Tue, 18 Mar 2025 06:11:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B1EA228000B; Tue, 18 Mar 2025 02:10:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ACEAC280007; Tue, 18 Mar 2025 02:10:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9BD6328000B; Tue, 18 Mar 2025 02:10:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 79A08280007 for ; Tue, 18 Mar 2025 02:10:46 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id DB24A1A06C1 for ; Tue, 18 Mar 2025 06:10:47 +0000 (UTC) X-FDA: 83233648134.22.0F74301 Received: from dggsgout11.his.huawei.com (dggsgout11.his.huawei.com [45.249.212.51]) by imf21.hostedemail.com (Postfix) with ESMTP id 111001C000F for ; Tue, 18 Mar 2025 06:10:44 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; spf=pass (imf21.hostedemail.com: domain of shikemeng@huaweicloud.com designates 45.249.212.51 as permitted sender) smtp.mailfrom=shikemeng@huaweicloud.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742278245; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vpEU5Buhndiapz3t4dOCy2d15De3El1V/w8gDaCFAGs=; b=FgtAesE8iimHcPGQOgWtFpUykPKSvemIikowyD6H1IUaJbazyD2V8xZJVK0+wwpqpPA4dv lXnZS0u/RqBFatsbCwcKFSJ7KVVnH2YL7NMWaOD6hz44eAiLaWFjglTGhjttKcx3KXzXmm F+wz76pjv0zzV/MOlxHrk4Zb1G7sJZY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742278245; a=rsa-sha256; cv=none; b=1J9JFGP7BjKuQkFivsXfhPjgYtRTPZ1rU7p1uh3xDlFAj73sHDjLvkkobPPMfq/uhNmclx GyP/XTpAJIBdpszY5GXOXpS5P0WQoKJqiqzcFkf0lSChe3lsKGg91WGay81VThOswJrxJn +TeuSFPvbuOv3iD3cVY8tP8KoZ5lKL8= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; spf=pass (imf21.hostedemail.com: domain of shikemeng@huaweicloud.com designates 45.249.212.51 as permitted sender) smtp.mailfrom=shikemeng@huaweicloud.com; dmarc=none Received: from mail.maildlp.com (unknown [172.19.163.216]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4ZH1f836FCz4f3lgR for ; Tue, 18 Mar 2025 14:10:08 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.75]) by mail.maildlp.com (Postfix) with ESMTP id 863D41A01A1 for ; Tue, 18 Mar 2025 14:10:32 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.101.6]) by APP2 (Coremail) with SMTP id Syh0CgD3AGRVDtln65ErGw--.37695S8; Tue, 18 Mar 2025 14:10:32 +0800 (CST) From: Kemeng Shi To: akpm@linux-foundation.org Cc: tim.c.chen@linux.intel.com, ryncsn@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 6/8] mm: swap: free each cluster individually in swap_entries_put_map_nr() Date: Tue, 18 Mar 2025 23:06:12 +0800 Message-Id: <20250318150614.6415-7-shikemeng@huaweicloud.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20250318150614.6415-1-shikemeng@huaweicloud.com> References: <20250318150614.6415-1-shikemeng@huaweicloud.com> MIME-Version: 1.0 X-CM-TRANSID: Syh0CgD3AGRVDtln65ErGw--.37695S8 X-Coremail-Antispam: 1UD129KBjvJXoWxGry5Jw13Kr4xGr48CF17Wrg_yoWruFyrpF yagrn8Krs7Xr43Jr4xJw4DZrWrZ3yvgF1Uta47Gr1SywnxJr1rWFyvy3ySgFyUC34kur90 y3W7K347uFs0qr7anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUPab4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M280x2IEY4vEnII2IxkI6r1a6r45M2 8IrcIa0xkI8VA2jI8067AKxVWUAVCq3wA2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAv FVAK0II2c7xJM28CjxkF64kEwVA0rcxSw2x7M28EF7xvwVC0I7IYx2IY67AKxVW7JVWDJw A2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxVW8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_GcCE 3s1l84ACjcxK6I8E87Iv6xkF7I0E14v26rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr2 1l5I8CrVACY4xI64kE6c02F40Ex7xfMcIj6xIIjxv20xvE14v26r1j6r18McIj6I8E87Iv 67AKxVWUJVW8JwAm72CE4IkC6x0Yz7v_Jr0_Gr1lF7xvr2IYc2Ij64vIr41lc7CjxVAaw2 AFwI0_JF0_Jw1l42xK82IYc2Ij64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAq x4xG67AKxVWUJVWUGwC20s026x8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r126r 1DMIIYrxkI7VAKI48JMIIF0xvE2Ix0cI8IcVAFwI0_JFI_Gr1lIxAIcVC0I7IYx2IY6xkF 7I0E14v26r4j6F4UMIIF0xvE42xK8VAvwI8IcIk0rVWUJVWUCwCI42IY6I8E87Iv67AKxV WUJVW8JwCI42IY6I8E87Iv6xkF7I0E14v26r4j6r4UJbIYCTnIWIevJa73UjIFyTuYvjxU s3kuDUUUU X-CM-SenderInfo: 5vklyvpphqwq5kxd4v5lfo033gof0z/ X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 111001C000F X-Stat-Signature: w9pugziwwosodukjyrga1n9pncawqtrs X-HE-Tag: 1742278244-659098 X-HE-Meta: U2FsdGVkX1+KDt8OCZKmPdwQoVkIdUzK452TSgclEEHs8Nq9upufjQ6T4fOT86y8SjXqRn2cfLfVVrSxHSPKQCPAYAr+E0Uc2t4+YIW4qEOP67qRvnBnqOaTT/8jpH1rRSUaQ8BpF6iEwt3l7BWXYw0saq3E0kdGMQILV0l6BX0ajwEaGQrhbTzT00ZwgPt9h9GZXmt3GZUA10wntoA6yPSzpWMy66ScbIQaKPLitLkTUnmiXEP/PxfjOcroQfbTJK+9/PESDSBpXMO3xjS86U/X89ChZ34BLtXiVWSxBc4MgBITpQUeYXo9iAmXiVfmRJN8Au3zmbL4tnV8T1tQNq/iRWh0Sk/Jn4+Hh79/PQHZKI60ybb04WlwApHOOvjnvghnogV5EYdamDIlMroOxSvREpAPF8va1tJRVbsmd23tsZ5Jlx0JBOtsj49XhGkTWM+TWnvd1XSefUXtlKO4U/JPVNAAxUmlgv8/7G85+G+AyuYIIsjeL6gFr5mee33IN8vYvBMorLOd2K1YVOyvOFXYLviLQhdxBKM0ulULDzVtjTMaWd5+RIcu8/FkXNMGhqmbmxms3dn2gjL9jyb2ZRGJ5wZ2TzUnCTJwdKjywb8D9cKQAmmRwRj1BQF0C//XpIIBvRQX54rMzxdCe5L0JQddaD5W9avEhxo6D5gqrLvHO5hghafiCXeGWos54i0OlwzAxbVDLGoHuSUK40JvCTD80YWCSU0EWKrj86wwiiTE4eN0E49RgrJfcagvJ0V1LhKEe68eeSy7TDYiDbFqpmmwg3c3AlAagWT1CKsR8a1ryVbV3ukj3uC9kzHndx7/f27fDC3W9DFzgyQaSvl0i7C0PjMH1eeliHIk9oFQ1IV/CDlFY62WkLKbrg3OJBMVA5i0GiEeDfLR4OtSzZGiOelrX+QK+XBO+39ATl0crNKeTkTOcljAyhfEwUKe7OAAEYQlLGKtGhbAIO45dBe t0E5hc+/ V2f754sFRNnewx41jVvZr0+VtSYqT/Abbdp1SMpqfABSmR1xMy5GL/J48iQfMGkB38izDIew9Nn/40AcXRW34/JZYhgdYC8PUDfN8pjHKN2lFBSEMkDFPxgsFZsekWpU0MXVM3TUw9rCPwutiwMPeY5o1kSCjr9BPiRCp1LB4lkpui5Dq0Xn+IdSC7dtNs3hRo4vwyXco291BUifv4CJkcBaSNA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: 1. Factor out general swap_entries_put_map() helper to drop entries belong to one cluster. If entries are last map, free entries in batch, otherwise put entries with cluster lock acquired and released only once. 2. Iterate and call swap_entries_put_map() for each cluster in swap_entries_put_nr() to leverage batch-remove for last map belong to one cluster and reduce lock acquire/release in fallback case. 3. As swap_entries_put_nr() won't handle SWAP_HSA_CACHE drop, rename it to swap_entries_put_map_nr(). 4. As we won't drop each entry invidually with swap_entry_put() now, do reclaim in free_swap_and_cache_nr() is because swap_entries_put_map_nr() is general routine to drop reference and the relcaim work should only be done in free_swap_and_cache_nr(). Remove stale comment accordingly. Signed-off-by: Kemeng Shi --- mm/swapfile.c | 70 +++++++++++++++++++++++---------------------------- 1 file changed, 32 insertions(+), 38 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index 134b061ef1ae..370509eb2f1f 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1455,25 +1455,10 @@ struct swap_info_struct *get_swap_device(swp_entry_t entry) return NULL; } -static unsigned char swap_entry_put(struct swap_info_struct *si, - swp_entry_t entry) -{ - struct swap_cluster_info *ci; - unsigned long offset = swp_offset(entry); - unsigned char usage; - - ci = lock_cluster(si, offset); - usage = swap_entry_put_locked(si, ci, entry, 1); - unlock_cluster(ci); - - return usage; -} - -static bool swap_entries_put_nr(struct swap_info_struct *si, - swp_entry_t entry, int nr) +static bool swap_entries_put_map(struct swap_info_struct *si, + swp_entry_t entry, int nr) { unsigned long offset = swp_offset(entry); - unsigned int type = swp_type(entry); struct swap_cluster_info *ci; bool has_cache = false; unsigned char count; @@ -1484,14 +1469,10 @@ static bool swap_entries_put_nr(struct swap_info_struct *si, count = swap_count(data_race(si->swap_map[offset])); if (count != 1 && count != SWAP_MAP_SHMEM) goto fallback; - /* cross into another cluster */ - if (nr > SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER) - goto fallback; ci = lock_cluster(si, offset); if (!swap_is_last_map(si, offset, nr, &has_cache)) { - unlock_cluster(ci); - goto fallback; + goto locked_fallback; } if (!has_cache) swap_entries_free(si, ci, entry, nr); @@ -1503,15 +1484,34 @@ static bool swap_entries_put_nr(struct swap_info_struct *si, return has_cache; fallback: - for (i = 0; i < nr; i++) { - if (data_race(si->swap_map[offset + i])) { - count = swap_entry_put(si, swp_entry(type, offset + i)); - if (count == SWAP_HAS_CACHE) - has_cache = true; - } else { - WARN_ON_ONCE(1); - } + ci = lock_cluster(si, offset); +locked_fallback: + for (i = 0; i < nr; i++, entry.val++) { + count = swap_entry_put_locked(si, ci, entry, 1); + if (count == SWAP_HAS_CACHE) + has_cache = true; + } + unlock_cluster(ci); + return has_cache; + +} + +static bool swap_entries_put_map_nr(struct swap_info_struct *si, + swp_entry_t entry, int nr) +{ + int cluster_nr, cluster_rest; + unsigned long offset = swp_offset(entry); + bool has_cache = false; + + cluster_rest = SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER; + while (nr) { + cluster_nr = min(nr, cluster_rest); + has_cache |= swap_entries_put_map(si, entry, cluster_nr); + cluster_rest = SWAPFILE_CLUSTER; + nr -= cluster_nr; + entry.val += cluster_nr; } + return has_cache; } @@ -1798,7 +1798,7 @@ void free_swap_and_cache_nr(swp_entry_t entry, int nr) /* * First free all entries in the range. */ - any_only_cache = swap_entries_put_nr(si, entry, nr); + any_only_cache = swap_entries_put_map_nr(si, entry, nr); /* * Short-circuit the below loop if none of the entries had their @@ -1808,13 +1808,7 @@ void free_swap_and_cache_nr(swp_entry_t entry, int nr) goto out; /* - * Now go back over the range trying to reclaim the swap cache. This is - * more efficient for large folios because we will only try to reclaim - * the swap once per folio in the common case. If we do - * swap_entry_put() and __try_to_reclaim_swap() in the same loop, the - * latter will get a reference and lock the folio for every individual - * page but will only succeed once the swap slot for every subpage is - * zero. + * Now go back over the range trying to reclaim the swap cache. */ for (offset = start_offset; offset < end_offset; offset += nr) { nr = 1;