From patchwork Thu Mar 20 11:48:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kemeng Shi X-Patchwork-Id: 14023363 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C5A0C36002 for ; Thu, 20 Mar 2025 02:53:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 81870280015; Wed, 19 Mar 2025 22:52:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7A018280017; Wed, 19 Mar 2025 22:52:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3593D280015; Wed, 19 Mar 2025 22:52:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 9E84D280016 for ; Wed, 19 Mar 2025 22:52:58 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 6AFAFC09C0 for ; Thu, 20 Mar 2025 02:52:58 +0000 (UTC) X-FDA: 83240407236.26.D6544CF Received: from dggsgout11.his.huawei.com (dggsgout11.his.huawei.com [45.249.212.51]) by imf17.hostedemail.com (Postfix) with ESMTP id B3ED640002 for ; Thu, 20 Mar 2025 02:52:54 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf17.hostedemail.com: domain of shikemeng@huaweicloud.com designates 45.249.212.51 as permitted sender) smtp.mailfrom=shikemeng@huaweicloud.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1742439176; a=rsa-sha256; cv=none; b=VlouvArnE/kI9oMAd680T+2/2uTgGg+NN5/s9RqGgHiLUWvUJ2WYIZeTSRcVttJp695XXh dyidnVmfGQPCbx1nHGJ6vRfPNsQz7llYSgrkppLDvHuOGMYz1SjSKspE2slO/BkftQaf3t lV35Ux5tOa66abcADJ5XxxsmbossS18= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf17.hostedemail.com: domain of shikemeng@huaweicloud.com designates 45.249.212.51 as permitted sender) smtp.mailfrom=shikemeng@huaweicloud.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1742439176; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QH6dKgQAdqVdSvPdFPWwhbGFKwf5MGYy6/tO3R6VlHE=; b=iR+CVLIz9u0CoAT7QDGaay41o0QsHwTuwNicdk0nQ3T1CAedwLLBHdcbQnLKQcPzdX+YBO AsNQopd0pUofbfuqBtazG8jEPT9m6vXJXkjtSlfue1w+CKdBofPfrFE7qYUDEtqyNQvgXZ vLfnZnC+Hxggr1T4KjsKItz3qY3pmGE= Received: from mail.maildlp.com (unknown [172.19.163.235]) by dggsgout11.his.huawei.com (SkyGuard) with ESMTP id 4ZJ9964x0qz4f3lWF for ; Thu, 20 Mar 2025 10:52:26 +0800 (CST) Received: from mail02.huawei.com (unknown [10.116.40.252]) by mail.maildlp.com (Postfix) with ESMTP id D3A7D1A058E for ; Thu, 20 Mar 2025 10:52:50 +0800 (CST) Received: from huaweicloud.com (unknown [10.175.101.6]) by APP3 (Coremail) with SMTP id _Ch0CgAHW8X_gttnYGGmGw--.42197S8; Thu, 20 Mar 2025 10:52:50 +0800 (CST) From: Kemeng Shi To: akpm@linux-foundation.org Cc: kasong@tencent.com, tim.c.chen@linux.intel.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 6/8] mm: swap: free each cluster individually in swap_entries_put_map_nr() Date: Thu, 20 Mar 2025 19:48:27 +0800 Message-Id: <20250320114829.25751-7-shikemeng@huaweicloud.com> X-Mailer: git-send-email 2.30.0 In-Reply-To: <20250320114829.25751-1-shikemeng@huaweicloud.com> References: <20250320114829.25751-1-shikemeng@huaweicloud.com> MIME-Version: 1.0 X-CM-TRANSID: _Ch0CgAHW8X_gttnYGGmGw--.42197S8 X-Coremail-Antispam: 1UD129KBjvJXoWxGry5Jw13Kr4xGr48CF17Wrg_yoWrtr4UpF yagrn8Krs7Xr43Jr4xJw4DZrWru3ykWF1Uta47Gr1SywnxCr1rWFyvy3ySgFyUC34kur90 y3W7K347uFs0qr7anT9S1TB71UUUUU7qnTZGkaVYY2UrUUUUjbIjqfuFe4nvWSU5nxnvy2 9KBjDU0xBIdaVrnRJUUUPab4IE77IF4wAFF20E14v26rWj6s0DM7CY07I20VC2zVCF04k2 6cxKx2IYs7xG6rWj6s0DM7CIcVAFz4kK6r1j6r18M280x2IEY4vEnII2IxkI6r1a6r45M2 8IrcIa0xkI8VA2jI8067AKxVWUAVCq3wA2048vs2IY020Ec7CjxVAFwI0_Xr0E3s1l8cAv FVAK0II2c7xJM28CjxkF64kEwVA0rcxSw2x7M28EF7xvwVC0I7IYx2IY67AKxVWDJVCq3w A2z4x0Y4vE2Ix0cI8IcVCY1x0267AKxVW8Jr0_Cr1UM28EF7xvwVC2z280aVAFwI0_GcCE 3s1l84ACjcxK6I8E87Iv6xkF7I0E14v26rxl6s0DM2AIxVAIcxkEcVAq07x20xvEncxIr2 1l5I8CrVACY4xI64kE6c02F40Ex7xfMcIj6xIIjxv20xvE14v26r1j6r18McIj6I8E87Iv 67AKxVWUJVW8JwAm72CE4IkC6x0Yz7v_Jr0_Gr1lF7xvr2IYc2Ij64vIr41lc7CjxVAaw2 AFwI0_JF0_Jw1l42xK82IYc2Ij64vIr41l4I8I3I0E4IkC6x0Yz7v_Jr0_Gr1lx2IqxVAq x4xG67AKxVWUJVWUGwC20s026x8GjcxK67AKxVWUGVWUWwC2zVAF1VAY17CE14v26r126r 1DMIIYrxkI7VAKI48JMIIF0xvE2Ix0cI8IcVAFwI0_JFI_Gr1lIxAIcVC0I7IYx2IY6xkF 7I0E14v26r4j6F4UMIIF0xvE42xK8VAvwI8IcIk0rVWUJVWUCwCI42IY6I8E87Iv67AKxV WUJVW8JwCI42IY6I8E87Iv6xkF7I0E14v26r4j6r4UJbIYCTnIWIevJa73UjIFyTuYvjxU s3kuDUUUU X-CM-SenderInfo: 5vklyvpphqwq5kxd4v5lfo033gof0z/ X-Rspamd-Server: rspam07 X-Rspam-User: X-Stat-Signature: g3u63do9g8z3e1it6wyif6r5porwx4h1 X-Rspamd-Queue-Id: B3ED640002 X-HE-Tag: 1742439174-709141 X-HE-Meta: U2FsdGVkX1+E7LQuteVM8I7VUyu0LArrM/l5qvjqOKU5huCWCF8HQbhH/4RRKwJlrKcz9tdAU+HqEKf1FUmgv5r5k0XUBLk6GVi2odHJ25LWKUmvRlALFMJFPbZfVT5fjvPKWvqBsjxTQR/Cw3iczG+1CtIf/CLrgDuYQbbRekgXKi0O+0FaSEXCOR86Bp/gB7HeGrHY9o+KVndWWpwMtXqOtT6/t/z0GwdDenS4nW9IzAFIgdvD9p9XEkkvTYe+5IGZzPVkZYMFj2oMadA/KAsaja6zql1rqLCCL2r5/hd71OeSoLtVL9+FKSpf60lOJ2cwpqljmOuId8vkx1kbxCb8Z3Ftvu2FCxzosKhf/5Vc+UE978aWOa+V3uMz08xOrem4itmEponCxLHeLyzjbOs5OMBiSTtevE5uPbO93dugNyPCV3qc7LjwNdPJFvKTbHyVr/yPqW3unjjNf77TPX1UHv9D6nzhI36Re+W6gTwQF00MCU7G0h0bbVsclrPzWeitcHor4CzvHxg+o6H6N5RgXOGvQdMQcNXLMufPrUjHyT3FeW6ib2XnT2o/NQ/JDElYLNI2nKOp9VPSlOvQnIKR6buI8DXpApixCuki00jIbyagmTy2QRcnZDss1hkwzBEAWh7s4hMVz8Cmr1aYifnets2mRXz9colawUBlEiqCbhzVSM9jMyAYk98WQnRwGGyJ3MLahzxNK6H177N9ZsLM+uhqy8oaW8mnq+TfKcxGmdrDhtQfibzqF/mFsWoYYFfFr/wlZqUTv8ftfmpNF9qZ1r1VFeMWRCaVbVQRwxCtJ2ymZDgYD4kGfEJWo4ukSXx6eQ0lIGxlO0SKTexIfkWdOwoxNdRQT1/Lj0AywTxuQ6YhlZ7HOj5Ri42iD5y22P4N+eP4v6RO4o0yV7mcNpGpEzMssah+h8rTr+32XOg4WJEOrQz9DsE/OmYm5HcUU8Zb4UuZuUA+aCPdKxo ye7B9h3l Qs4z7/DfnqJQlnttnBUkHrNLurfBKu+gZSJ7ezlYjLjPu7NrSu9um/4UljdvnUoIFj+9S128YH90EDE7oS5kwPbCp1btUFU+KqFnQbEizi3PeO8h4mJ4p+CRyPj7vI6+1Kvn5rHHkNr8NgaIqrqmN+1qa2rQy2N/naz6iBh8BuwCJSTH093VXmhpJ0gg0ZsJT0OZ1eHSp/XVvSpD2Hw4tY6AFvag7sGPYkcTz4XAydmKXkxA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: 1. Factor out general swap_entries_put_map() helper to drop entries belong to one cluster. If entries are last map, free entries in batch, otherwise put entries with cluster lock acquired and released only once. 2. Iterate and call swap_entries_put_map() for each cluster in swap_entries_put_nr() to leverage batch-remove for last map belong to one cluster and reduce lock acquire/release in fallback case. 3. As swap_entries_put_nr() won't handle SWAP_HSA_CACHE drop, rename it to swap_entries_put_map_nr(). 4. As we won't drop each entry invidually with swap_entry_put() now, do reclaim in free_swap_and_cache_nr() is because swap_entries_put_map_nr() is general routine to drop reference and the relcaim work should only be done in free_swap_and_cache_nr(). Remove stale comment accordingly. Signed-off-by: Kemeng Shi Reviewed-by: Tim Chen --- mm/swapfile.c | 70 +++++++++++++++++++++++---------------------------- 1 file changed, 32 insertions(+), 38 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index 6f11619665e8..646efccdd2ec 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1455,25 +1455,10 @@ struct swap_info_struct *get_swap_device(swp_entry_t entry) return NULL; } -static unsigned char swap_entry_put(struct swap_info_struct *si, - swp_entry_t entry) -{ - struct swap_cluster_info *ci; - unsigned long offset = swp_offset(entry); - unsigned char usage; - - ci = lock_cluster(si, offset); - usage = swap_entry_put_locked(si, ci, entry, 1); - unlock_cluster(ci); - - return usage; -} - -static bool swap_entries_put_nr(struct swap_info_struct *si, - swp_entry_t entry, int nr) +static bool swap_entries_put_map(struct swap_info_struct *si, + swp_entry_t entry, int nr) { unsigned long offset = swp_offset(entry); - unsigned int type = swp_type(entry); struct swap_cluster_info *ci; bool has_cache = false; unsigned char count; @@ -1484,14 +1469,10 @@ static bool swap_entries_put_nr(struct swap_info_struct *si, count = swap_count(data_race(si->swap_map[offset])); if (count != 1 && count != SWAP_MAP_SHMEM) goto fallback; - /* cross into another cluster */ - if (nr > SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER) - goto fallback; ci = lock_cluster(si, offset); if (!swap_is_last_map(si, offset, nr, &has_cache)) { - unlock_cluster(ci); - goto fallback; + goto locked_fallback; } if (!has_cache) swap_entries_free(si, ci, entry, nr); @@ -1503,15 +1484,34 @@ static bool swap_entries_put_nr(struct swap_info_struct *si, return has_cache; fallback: - for (i = 0; i < nr; i++) { - if (data_race(si->swap_map[offset + i])) { - count = swap_entry_put(si, swp_entry(type, offset + i)); - if (count == SWAP_HAS_CACHE) - has_cache = true; - } else { - WARN_ON_ONCE(1); - } + ci = lock_cluster(si, offset); +locked_fallback: + for (i = 0; i < nr; i++, entry.val++) { + count = swap_entry_put_locked(si, ci, entry, 1); + if (count == SWAP_HAS_CACHE) + has_cache = true; + } + unlock_cluster(ci); + return has_cache; + +} + +static bool swap_entries_put_map_nr(struct swap_info_struct *si, + swp_entry_t entry, int nr) +{ + int cluster_nr, cluster_rest; + unsigned long offset = swp_offset(entry); + bool has_cache = false; + + cluster_rest = SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER; + while (nr) { + cluster_nr = min(nr, cluster_rest); + has_cache |= swap_entries_put_map(si, entry, cluster_nr); + cluster_rest = SWAPFILE_CLUSTER; + nr -= cluster_nr; + entry.val += cluster_nr; } + return has_cache; } @@ -1806,7 +1806,7 @@ void free_swap_and_cache_nr(swp_entry_t entry, int nr) /* * First free all entries in the range. */ - any_only_cache = swap_entries_put_nr(si, entry, nr); + any_only_cache = swap_entries_put_map_nr(si, entry, nr); /* * Short-circuit the below loop if none of the entries had their @@ -1816,13 +1816,7 @@ void free_swap_and_cache_nr(swp_entry_t entry, int nr) goto out; /* - * Now go back over the range trying to reclaim the swap cache. This is - * more efficient for large folios because we will only try to reclaim - * the swap once per folio in the common case. If we do - * swap_entry_put() and __try_to_reclaim_swap() in the same loop, the - * latter will get a reference and lock the folio for every individual - * page but will only succeed once the swap slot for every subpage is - * zero. + * Now go back over the range trying to reclaim the swap cache. */ for (offset = start_offset; offset < end_offset; offset += nr) { nr = 1;