From patchwork Wed May 29 08:28:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13678362 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C88EFC25B75 for ; Wed, 29 May 2024 08:28:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 575B26B009B; Wed, 29 May 2024 04:28:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4FE4D6B009D; Wed, 29 May 2024 04:28:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 378376B009E; Wed, 29 May 2024 04:28:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 16ACB6B009B for ; Wed, 29 May 2024 04:28:54 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B5B3180825 for ; Wed, 29 May 2024 08:28:53 +0000 (UTC) X-FDA: 82170757746.29.DA9D04E Received: from mail-oa1-f49.google.com (mail-oa1-f49.google.com [209.85.160.49]) by imf30.hostedemail.com (Postfix) with ESMTP id E3CF78000A for ; Wed, 29 May 2024 08:28:51 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=fRBHjD75; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf30.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.160.49 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1716971331; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7k7udukJq/5lufC/QvLOUP0N512BAUWebXH46Q8HtFk=; b=bf60o0PaGw4ekw18Kie4yl7PiAfhNTlTQ9IkCvHv8bC42V4GK0BE9U8GT2fS3amxDJF4z+ VQLZg2noWkrCIa2l8tDjm4vTa1c71Bmd1pMR1xWgUXsc7HwekmG+Bgl520B6LbOTKtOpxU J1KtV54EZTIS+pLA2yA+3KRy3ej55KQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1716971331; a=rsa-sha256; cv=none; b=l/eR9qAmpa30gxqREtRvzlZ+KBXukBc0mSN2LoIIFdhbGU3IWhmPS0F6hAg02ppLIx2GPj ntokW0EKYKEUaZavZbgjoQwPQk6lum/zAkte1ppZ0D8iNGiccfcbqKKft3Ijj2qjeXYtqi fYv0QmO0iOROOcwWRDNAYgOljegyXxY= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=fRBHjD75; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf30.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.160.49 as permitted sender) smtp.mailfrom=21cnbao@gmail.com Received: by mail-oa1-f49.google.com with SMTP id 586e51a60fabf-24c9f2b7b19so873730fac.0 for ; Wed, 29 May 2024 01:28:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716971331; x=1717576131; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7k7udukJq/5lufC/QvLOUP0N512BAUWebXH46Q8HtFk=; b=fRBHjD756wIF8FRj9AacI9sgkaq0awWGKeeIwK//eoi1Ncbe33MFK1buB5Z5xnSb8n fT8B4YBejVYCXT0yqsY6atuK1GB4Trb2gNDb0cJ2PzfrtRuFTRj3EIfx88MFycHHEPUb NgeNuA4KYgL+A5rdfAhgY0GFsNckKcMaBmgoPTSlWVAiRDDKTbuYK/GkSv1Q+i6JpixO vp7HwS91bOVAD4W9NLRtmvqfWd0ZTwpnEZ+Gnp37xIsb8o7AsPtNk7ycCcFI0/p6gtJS t7k98IR5tXP/QMNhmJ3yvknckJzObI49tDpM0V/TRoHSXt5a+Yrg0NarjhdCGzm15acl q8kg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716971331; x=1717576131; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7k7udukJq/5lufC/QvLOUP0N512BAUWebXH46Q8HtFk=; b=VOb/w0g8vD3rj16vOLsQzACExYUhfktVEmb7omV6yPYkg5TK20xLf2OvGnjn7r+qzx B69T4mUtjzEggZjlx+FhXb33T/xDTOeUkMRx5UgpYMKnV6fM3NeOxfR++2wlGQcd/TvA upe16Zv2W8RpOCzxuadRH8nEoA7+C/BX07e2eqAqI0k+vkosuF/Psv+0ftCCImHRUFUJ BIc7FL9xXSTTguwYNlNEjBuM3vM440RhPlP3nJciiNEAPUO/306SSF18igQ58/Lsdix+ +X+B8kIVeSiAJgOQf3UZ02aictJtBjCG9Bvw3Rf1SeXKJrKQ0w3g5Azr/MGVUK5m9FwK P0jg== X-Forwarded-Encrypted: i=1; AJvYcCVURSTZjkhhQRCxUgoRpRACXy4W1GnM5OnXkstK5m/BBKfalIJ45v95p+fwx360VuzsBdgt92m5f0NuhE+Ko04PBJ8= X-Gm-Message-State: AOJu0YzJQFYKweZWT3SpIHgLKwASVFKo8hvUxcPp7LTHVsl5a7FYqt7a lv0U/gju/Y2gQmV6teEE4502ygvsRQEXzeCMihnvBdps6a5GHbQQ X-Google-Smtp-Source: AGHT+IFwms1cBvU+f9ojhWDg3nZKfLTfOVGOcCh0o9BJomwXX1OgvRueLY1XlvYRSI7zo2uCYFeifw== X-Received: by 2002:a05:6870:96a2:b0:24c:a414:3eab with SMTP id 586e51a60fabf-24ca41467f3mr15324961fac.32.1716971330911; Wed, 29 May 2024 01:28:50 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-702220d948bsm174916b3a.131.2024.05.29.01.28.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 May 2024 01:28:50 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v5 1/6] mm: swap: introduce swap_free_nr() for batched swap_free() Date: Wed, 29 May 2024 20:28:19 +1200 Message-Id: <20240529082824.150954-2-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240529082824.150954-1-21cnbao@gmail.com> References: <20240529082824.150954-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Stat-Signature: sp7o6wsfe4apd8xrsr8b6o4nqzn77f41 X-Rspamd-Queue-Id: E3CF78000A X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1716971331-728022 X-HE-Meta: U2FsdGVkX19035okKhzh6yNI8Osgogn/u4TjoWdPR6upa5naWIzqDPHK7zX7EwhsD8mcyUa41wD2eWYDmGj85QmciYSo7mQPQ8ba3jWPo993fnKShOUZaL71qoX7pOUYnRO2tRyQx8azfI8Mg8Ud3KAn1ZqDk3jpeeiGDXL3QwnqnVFcLGbwjCfE4zoFk+DWEin4V+R5fT+G7iOee2ZGosb4bOgjYsHjFzUICSDX75eBreT3+5pFuAPc4v9ayuiUfilhalwftqMXPhcw4GbL3KytTMOAh93+e4CKgtVgKu0Qqu3hBcRzm+AyC3tD5w6AjG2+D0WWLZnItgYu+ycPFlGf8p2p/isvUVnKwHLjo0s7u3OpoxV2/pNWx+UdxUDljvzuwsPKouk6+M0SvzS5yR4ZOBcUeq0eqNQhy/qHnxjbitGBteaFIyu4t2FJDc/q2MDfv6R7kQkTxmF2V/BnfvCavQTTcdyyiFjAxB8UZufieboJJOHnnLeTfsHBSuExmQhdEo9kyjC3BEzn89ffec7GvvxkYR05fhlS295bg8Ylluje6dcIxigtHAqQ7UJqei/qQjDvY07hd8xecsyPP/g9+Wodn90mjqzxmr+E1+m9n1T136/9JpQO+PVz3vvujhxgNL3RSbLLTxJGl1eMCAHV1WtOSnEb9uW4afX3BTwpxEA6rfim+kki5QYWssPKpKjLOjoMvxGLbhheqXC4p97cswoa+SOrahKhXtw3Y7duKRkwEG1Hlho2WWvsrrr44Y5UkK3p3aWED0u0H7gOVM2LxAToaajxISzZU9GfCIQt31iHs8+eFoaWCyC+dFm3LgDfHHmbrWLzTuAml14aX3V8FiseV4jUTqcU/rlN0dq2Io1NYuXWdiO2fOsvx3ViB3Wg/d5Wk/Zzzd+mqg1ubax86cqof5ammIhChB9f8H46sIxsmUtkdMmnFzb0HSmFkfwz0wOhgt5YltunRjO VKPupbEb 2j53dPkY7oCVwLqFKMfwS1eWI5zyTYQfZ4wSg/xxXjyZoHlnHlbixn1tBtVvrp7gj8//Gq5hFBSXx74mYrNxNW04Cvz8wan3Gzc5arfuliUQUnrZA2Oku65uM6xA3Q4vTuIAmO072jTV/Azv5sJiyQZYDKUwRPYws3Hs/slwB7LH/qrr4QhuzJFzOCVn9c5RJXCUcsuWnCsU2NT/PZyUWSIVBHT3GI13iUlAWMVTP2isRS32hG4vr6BO5Y8jj+mDpy04GvFQg0PisH8bXxViNrOzCWZbQMAshbEllQmzQdW4p7zlQZDMcwCh3cNG/5QUODuo4ukJhwkfs+pLUg9TchzrBZF4E8F7QSyQifvgEU/9Qc/r/d/ts7g6B8Kt+/Gb2gVVZq5Iw4xuMaIYTpUpBf0yKre6mO7ZHQa4diwR4QMDnFUmnzYzmRB/fIcxP4dI/i6vmYEa6Vv0lTZmRhuepM/ptppffQJneZtzWeAvwY1JG6KdCa4k9kYlrxy0BQjj+8RihwoZNgbiiU0L89Ee5MeQ60W0GsB2nL8L9OXDfu8FtT49dTS9zmoqmzLv66h4myEUI8fCnj89A2ddtlJm/ZyJdSYAu7E0H2Hux7PT0AB8XhjQKrR5sc1nIjQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han While swapping in a large folio, we need to free swaps related to the whole folio. To avoid frequently acquiring and releasing swap locks, it is better to introduce an API for batched free. Furthermore, this new function, swap_free_nr(), is designed to efficiently handle various scenarios for releasing a specified number, nr, of swap entries. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Reviewed-by: Ryan Roberts Acked-by: Chris Li Reviewed-by: "Huang, Ying" --- include/linux/swap.h | 5 +++++ mm/swapfile.c | 47 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+) diff --git a/include/linux/swap.h b/include/linux/swap.h index a11c75e897ec..45f76dfe29b1 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -481,6 +481,7 @@ extern void swap_shmem_alloc(swp_entry_t); extern int swap_duplicate(swp_entry_t); extern int swapcache_prepare(swp_entry_t); extern void swap_free(swp_entry_t); +extern void swap_free_nr(swp_entry_t entry, int nr_pages); extern void swapcache_free_entries(swp_entry_t *entries, int n); extern void free_swap_and_cache_nr(swp_entry_t entry, int nr); int swap_type_of(dev_t device, sector_t offset); @@ -562,6 +563,10 @@ static inline void swap_free(swp_entry_t swp) { } +static inline void swap_free_nr(swp_entry_t entry, int nr_pages) +{ +} + static inline void put_swap_folio(struct folio *folio, swp_entry_t swp) { } diff --git a/mm/swapfile.c b/mm/swapfile.c index f1e559e216bd..92a045d34a97 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1356,6 +1356,53 @@ void swap_free(swp_entry_t entry) __swap_entry_free(p, entry); } +static void cluster_swap_free_nr(struct swap_info_struct *sis, + unsigned long offset, int nr_pages) +{ + struct swap_cluster_info *ci; + DECLARE_BITMAP(to_free, BITS_PER_LONG) = { 0 }; + int i, nr; + + ci = lock_cluster_or_swap_info(sis, offset); + while (nr_pages) { + nr = min(BITS_PER_LONG, nr_pages); + for (i = 0; i < nr; i++) { + if (!__swap_entry_free_locked(sis, offset + i, 1)) + bitmap_set(to_free, i, 1); + } + if (!bitmap_empty(to_free, BITS_PER_LONG)) { + unlock_cluster_or_swap_info(sis, ci); + for_each_set_bit(i, to_free, BITS_PER_LONG) + free_swap_slot(swp_entry(sis->type, offset + i)); + if (nr == nr_pages) + return; + bitmap_clear(to_free, 0, BITS_PER_LONG); + ci = lock_cluster_or_swap_info(sis, offset); + } + offset += nr; + nr_pages -= nr; + } + unlock_cluster_or_swap_info(sis, ci); +} + +void swap_free_nr(swp_entry_t entry, int nr_pages) +{ + int nr; + struct swap_info_struct *sis; + unsigned long offset = swp_offset(entry); + + sis = _swap_info_get(entry); + if (!sis) + return; + + while (nr_pages) { + nr = min_t(int, nr_pages, SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER); + cluster_swap_free_nr(sis, offset, nr); + offset += nr; + nr_pages -= nr; + } +} + /* * Called after dropping swapcache to decrease refcnt to swap entries. */ From patchwork Wed May 29 08:28:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13678363 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2265FC25B75 for ; Wed, 29 May 2024 08:29:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AA6AE6B009E; Wed, 29 May 2024 04:29:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A2E786B009F; Wed, 29 May 2024 04:29:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 881B66B00A0; Wed, 29 May 2024 04:29:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 6427F6B009E for ; Wed, 29 May 2024 04:29:02 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 21D5C1C1280 for ; Wed, 29 May 2024 08:29:02 +0000 (UTC) X-FDA: 82170758124.23.446E961 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by imf05.hostedemail.com (Postfix) with ESMTP id 4D62C100013 for ; Wed, 29 May 2024 08:29:00 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=AlgVVsfW; spf=pass (imf05.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1716971340; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=o8PkNiQW/mkvjIvpLEp+EBywhuNk7TaiPl+XTuB214I=; b=LV0J2Ys1Qw75Gx4OhgzCKSYI6WbDjnRtBQ5AdIOUPY4MH3yRFE+tDvFLdC+S4KSwh7jJnZ S9cVjo4w/l6jI3VQel3syaJSAGHrd1whk2gOioEg0G2uORvP3JLJs3joHvwTQGLt0Ut8v7 G20hnhl+zVNsnwWj2mrnRhlIdgcDYTw= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=AlgVVsfW; spf=pass (imf05.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1716971340; a=rsa-sha256; cv=none; b=A2LubWcLIEuKgbjJ1KRbe0KGqRP3yJs0LySHp4NoT5Q/O7+zrbv4NHg38jbvc+CH2xz2GO ox1OeSgWDggjZzmb9qUjyzXVRtjnYWYaudSOlgj08I8T6LTOslXJUqu+ZFJ36CYV4hV1/h +DtdFGwe3x8b0WoFkfsM01O8+WHVOGA= Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-6f966840af7so1522922b3a.2 for ; Wed, 29 May 2024 01:29:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716971339; x=1717576139; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=o8PkNiQW/mkvjIvpLEp+EBywhuNk7TaiPl+XTuB214I=; b=AlgVVsfWqb6vlqXPsumcxd/U7AmW8o70PNC8JSkBjJKtRSeKuZ7XknTzNp7A4omj5f hURdBgsBQE2tuN8EFuDpus6jzioWMhshAHd0tomTDfbljXJ870oiEhG9YeI1tA6OoC4F PtEoOBVrMJlWAoSchYfu9Zv34MHSmMTf9buZDMvS8eFT0ft1Vw79eafK0Wh1kyn7Sq+v uOkbYSbSX0Lv/S+FE2virtoSz6D12eHsl/eYc12I7Sg1pnsiKdyO6iKkjSwGccX1Oc3t 3NzqX0EqH9POPTlinP/cK8g1R5iSdUkek4A6xGsEwBhhTHLL1OA3Q69XVlcaQ7cXUya4 QOfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716971339; x=1717576139; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=o8PkNiQW/mkvjIvpLEp+EBywhuNk7TaiPl+XTuB214I=; b=XB3UcRUh4Tgh1ImTsFbxjfXY1N21HxX/hHqKKcFr5tJVdlLJZ1V802fqMUOwZzD452 upxCp1/owR5kHo1jsqLcmKV003DSy9GojorItYlYAAUluW/IAMDiXqvRAl/LPo2q2DEA 3QmhqbauWEvQX/Y/vh9DaMmJjLeBYCLuGVjHS+5EtCWebAdf0yTnT9qq+LcFfgDLOy73 YctOdxKIuDTZcbKnh7QVS+64Bd60Etvfgzto5zthl07/eeUAzq1SKJaBLM3U+DcJVpYg N7r65R1firHQwEvPnb9hF+gngpiEopaAIEGIHiz2v+RVVxzz2z2IHLV3MM2d/b7w2YkM 8dvA== X-Forwarded-Encrypted: i=1; AJvYcCVBbnACKwZLFPvEfUZebyxxHd7kEd5jmOhhzDgutwBzBLL21csUeMlPIQWHE65WNvw9fegCw5H/K67I88+9EVZxPlw= X-Gm-Message-State: AOJu0YwrcDtMLlceyjvw/oKHQXtnSh1KR39wg+zTCf54TrJUxGiPwtbs fFyKMP37WFG1gmkirpw4EwAivZpyBGewfKlb1ecWJWcbRB8zT+DO X-Google-Smtp-Source: AGHT+IFIIMwl+a1BFU1LURBAWUDnpnOzYB+DE372n3RJK0yLX+C7PJpG+VjhO1aGqsG0jKGc/xlpAg== X-Received: by 2002:a05:6a00:4405:b0:6f8:caf2:8f4f with SMTP id d2e1a72fcca58-6f8f45d2624mr16218543b3a.33.1716971338932; Wed, 29 May 2024 01:28:58 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-702220d948bsm174916b3a.131.2024.05.29.01.28.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 May 2024 01:28:58 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, "Rafael J. Wysocki" , Pavel Machek , Len Brown , Christoph Hellwig Subject: [PATCH v5 2/6] mm: remove the implementation of swap_free() and always use swap_free_nr() Date: Wed, 29 May 2024 20:28:20 +1200 Message-Id: <20240529082824.150954-3-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240529082824.150954-1-21cnbao@gmail.com> References: <20240529082824.150954-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Stat-Signature: x5cgfndziqo4fms9er6odggwwy7oxdhr X-Rspamd-Queue-Id: 4D62C100013 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1716971340-370988 X-HE-Meta: U2FsdGVkX1/kUqAYv0dNEA1AdTdwi+m+7/yedSLoFx9Myed4jBuZflZw9N5PkempZAcp0SKnXBrlhxfBGiuPTVNTJngnAJgjqu8tZk/a3QQTDlo42dwMInT4NurCDicwkf8TS5ghZNAigXqZl6X6Md1aL5RHYESN9Bcgon17PQD/qMSoznUY9fmSHqjqlT6KxlkvMsm7rhJ5IUw9nP8E5ASKaC0y0F62Uov7gaZnHuq5wfDP6SMseAFjV4W/zchazbzDXYpK/vJFDgEzzw2832WhczOgMc4WznvcTs0VkyXudUwxZf0l8tpnC8KDSi678b/ek0aPdXYuudyeT6FoKXZCkKvPUwsr85ycmArrNWMUbFmce7Zq4HBLYUNy41WovwzD4Mg1DzdJeTRAOKDShIdgHOpwJwkq1zwM45/JHgIQuVYL0WLIlmVXDQ2XoS2PJbsOM1DAIKO41bkk6eb1CF8vFWv9+vnB0JQ/qITZb+QvLo9BlSdrHaD/eTYndn4zJMjezvjWnePiyg27uaDnqjsRMXrX+l/y6BaUYxPMrmcAim2YTm/nsdJm4GU1ksQVQc74ZhnTnqUMeZm+1oIzb7OnyZ04+me0fnwD9mAX2T5V5kUy/CnDorFdVSbY5FPyRAWrt2oy02Fj5Hk/10Nj4BAmy2TxzMM3R9mBROVQq8HldYvB4UrBLKot7TZHlxEjYsAGmb6+tFb/h/x5qPmkL6fXW0Ty9OsBIdKWLiaeKeSFv+cjgf7RxFAUgMdSAwFm1oU33BPTitBVkVcBh9BQsbPEQ+D/sNUrAW+0fB16JYqdDb8Gi4O56F83OY/xhMUHVk5K+nn5VV/WHTfnMQetgA7chNUUTUV55ZzZL5kjvY3K9nJCfMJVKvthO2ZHezsi/qNy0/4DFN8ErYQQBIIa7P8Ufy8q60tqUpRpuDjmHk7eM+6QccRMhgUZZZIlyE2EYCO4F4Hn6Wi3y+iiDz/ AxbG+arr ggdIRxn5UKQfIBr9JoB7ioc8PqQSkk7xNULCeyg8pPsK70ocZ9G/IY8Z2qQsvZoxDgl8hR5VhI8FRUiPXvW20KVXIwdeIvIWzK1C0S0bcmhiGhnAsc1WVq/9aDJXOShLPS9keC1oqTIfcsDIq38byCv0KMQcTobpGrG94uUW8TeRZGW7xqEAafAviKJwrqKCVEyXQLzftdPFIiUimqMlIGtUdMD3+yMNCfUCTFR+17zkH5SS+kvU9a6IhFJN4pKmydtNPbdbz6yT1Ceggm+HU/IhbQreWzHrL+q/Z0yRrmdfFelWeU4xJ7PqOG/CY476LnHkogu2qQL1NacH0YkIIt0ZunXU0mxh1jLV3Tc6Jb12IN8/f2mqH7su9enUgURAhD66r0u0mdn+8fDwpqIfCnRQaVRDhxhs2BCfp/YlWe2YeagGVzLuXoLLbLakvK1w3oSk2bgefcXiDH0casSjkyIJA9lN+y/K695UXX+G9IuM810Yje0bthMQGmtGiC7iV5JL3dcrhSKh7bWdySovTDxfNsESHEJRnRo+gZ4v5pDGUToLRdrE0vO0xDV1YSX7ADdWirKAuXaGb7KtRzQOlJPgHDita9bSBDrQ72QoRkx6KtepOJUumhCpYOjAlaWm8frhbyCIUwK78b3P7iZGP7C1PqlJfjIRQmArsmt8ViK+cUJwCBSwfbbmMmQflt0/ENnxpltelhElP1QbMPf7wwcxkpInXD7mrUuhDtqW3rzID7p6xDz7OjigZXnU6CBxKEYtZlbhkrWKrNMl7kvoAGe5ms7umsS3X4Glpna0h+v0Pw00= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song To streamline maintenance efforts, we propose removing the implementation of swap_free(). Instead, we can simply invoke swap_free_nr() with nr set to 1. swap_free_nr() is designed with a bitmap consisting of only one long, resulting in overhead that can be ignored for cases where nr equals 1. A prime candidate for leveraging swap_free_nr() lies within kernel/power/swap.c. Implementing this change facilitates the adoption of batch processing for hibernation. Suggested-by: "Huang, Ying" Signed-off-by: Barry Song Reviewed-by: "Huang, Ying" Acked-by: Chris Li Reviewed-by: Ryan Roberts Cc: "Rafael J. Wysocki" Cc: Pavel Machek Cc: Len Brown Cc: Hugh Dickins Cc: Christoph Hellwig --- include/linux/swap.h | 10 +++++----- kernel/power/swap.c | 5 ++--- mm/swapfile.c | 17 ++++------------- 3 files changed, 11 insertions(+), 21 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 45f76dfe29b1..3df75d62a835 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -480,7 +480,6 @@ extern int add_swap_count_continuation(swp_entry_t, gfp_t); extern void swap_shmem_alloc(swp_entry_t); extern int swap_duplicate(swp_entry_t); extern int swapcache_prepare(swp_entry_t); -extern void swap_free(swp_entry_t); extern void swap_free_nr(swp_entry_t entry, int nr_pages); extern void swapcache_free_entries(swp_entry_t *entries, int n); extern void free_swap_and_cache_nr(swp_entry_t entry, int nr); @@ -559,10 +558,6 @@ static inline int swapcache_prepare(swp_entry_t swp) return 0; } -static inline void swap_free(swp_entry_t swp) -{ -} - static inline void swap_free_nr(swp_entry_t entry, int nr_pages) { } @@ -611,6 +606,11 @@ static inline void free_swap_and_cache(swp_entry_t entry) free_swap_and_cache_nr(entry, 1); } +static inline void swap_free(swp_entry_t entry) +{ + swap_free_nr(entry, 1); +} + #ifdef CONFIG_MEMCG static inline int mem_cgroup_swappiness(struct mem_cgroup *memcg) { diff --git a/kernel/power/swap.c b/kernel/power/swap.c index d9abb7ab031d..85a8b5f4a081 100644 --- a/kernel/power/swap.c +++ b/kernel/power/swap.c @@ -200,12 +200,11 @@ void free_all_swap_pages(int swap) while ((node = swsusp_extents.rb_node)) { struct swsusp_extent *ext; - unsigned long offset; ext = rb_entry(node, struct swsusp_extent, node); rb_erase(node, &swsusp_extents); - for (offset = ext->start; offset <= ext->end; offset++) - swap_free(swp_entry(swap, offset)); + swap_free_nr(swp_entry(swap, ext->start), + ext->end - ext->start + 1); kfree(ext); } diff --git a/mm/swapfile.c b/mm/swapfile.c index 92a045d34a97..9c6d8e557c0f 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1343,19 +1343,6 @@ static void swap_entry_free(struct swap_info_struct *p, swp_entry_t entry) swap_range_free(p, offset, 1); } -/* - * Caller has made sure that the swap device corresponding to entry - * is still around or has not been recycled. - */ -void swap_free(swp_entry_t entry) -{ - struct swap_info_struct *p; - - p = _swap_info_get(entry); - if (p) - __swap_entry_free(p, entry); -} - static void cluster_swap_free_nr(struct swap_info_struct *sis, unsigned long offset, int nr_pages) { @@ -1385,6 +1372,10 @@ static void cluster_swap_free_nr(struct swap_info_struct *sis, unlock_cluster_or_swap_info(sis, ci); } +/* + * Caller has made sure that the swap device corresponding to entry + * is still around or has not been recycled. + */ void swap_free_nr(swp_entry_t entry, int nr_pages) { int nr; From patchwork Wed May 29 08:28:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13678364 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75D5CC25B75 for ; Wed, 29 May 2024 08:29:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0458B6B00A0; Wed, 29 May 2024 04:29:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F11346B00A1; Wed, 29 May 2024 04:29:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D63726B00A3; Wed, 29 May 2024 04:29:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B1C496B00A0 for ; Wed, 29 May 2024 04:29:09 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6ED1A806EE for ; Wed, 29 May 2024 08:29:09 +0000 (UTC) X-FDA: 82170758418.17.1E9A5C5 Received: from mail-pf1-f170.google.com (mail-pf1-f170.google.com [209.85.210.170]) by imf07.hostedemail.com (Postfix) with ESMTP id AAFD440003 for ; Wed, 29 May 2024 08:29:07 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LVGBobmN; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf07.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.170 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1716971347; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=83K8bgM725IDCOF3GoIeML6r90yg5uTNSjeWCxUl1nI=; b=E8QuOBkKRUQg0OrNsH6RS0Ggbbq7I4EoVrGNaF/kZGdARzpShN8rfB8W4g9hJnh4eauChU XxhPKHpgVOrIKFc2KSuaDUSXhnE5v4DVHY53Mgg29F7xgPKdCz14OQ8PpzeWohPERsFmYn zdSPWd8SPzgXbNZrQNEc+YldZfaOm5Q= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1716971347; a=rsa-sha256; cv=none; b=koaG9zNUUfKgaMKhqbMU9dxQLmNUMvDdP9gh8Fjl0Z63jH+pHmbr7RuDib6cnKXdQ+M8/g +YwfUmKJ4cnHtlPjJFkhJDSgZfELf/dFGx3kI/f+cqex1e4/+UHkwsGnjO7D3KlQTywRkt zr/praqf6ysSnBhpdfm58qlsjVBGNE4= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=LVGBobmN; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf07.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.170 as permitted sender) smtp.mailfrom=21cnbao@gmail.com Received: by mail-pf1-f170.google.com with SMTP id d2e1a72fcca58-6f6bddf57f6so1775073b3a.0 for ; Wed, 29 May 2024 01:29:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716971346; x=1717576146; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=83K8bgM725IDCOF3GoIeML6r90yg5uTNSjeWCxUl1nI=; b=LVGBobmND3UybK3XFtObW8Muvq95st4KE75O/jWEyRJz8TkWqAWVK0akEU8gn8gu6Z AFfPN9G57dgnf4Jupy0VUwQkkA1u1iRJIegZK55EhIpaXq5CeFpPf3eUsipEGF+63Ysk DCCJwVZ6Aiier8KKgiu1aaR0J4nI7bBFbWsejqNpdAj8ZtjRsLP7HFDAxt9kgr5uRtkv t3ZZUj5AzSQZ8Mlq7cQC2LT7TH7VqIbubj+sEuV+AqMtIgsCTYr35wfWS3ptpDB2p3jW EccVWULl0JSGzwfG7CIKZS5vNl5zP4BafyZqt1oIOY9ZWZ99gZqrv/vMpE9u89Pb2gru TSAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716971346; x=1717576146; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=83K8bgM725IDCOF3GoIeML6r90yg5uTNSjeWCxUl1nI=; b=RehWcscRMlJjaLa4DdkZXvVfqch/zd4k41pTw7dCGAW/1TgKd5+zyqSfX3u3kOfIHr WPJAUQQfsT4VCQQZI7eR9TvOG81LZAxwmUzaV99IvhybCupW2d1iV26dRDJgYEBASYn/ soj4x9s/WBkf0Yh/bEfU+HgVe87ZKjB2r5NPlvpkR7+T0T6fNpllsx/ekcs7pOyP+dgS 2Nj5W/4OnZCW++fAbSFJVDRr3HgxwOWC3INvBVoQYdKe+xuUlblFjfqAIJxFZ+zRXYzG A7MDSczmwnqgvU6c1gnyHSavIe79EsupXqvCzOcDFkBCI0cVq15ZPO9Xd8qw7KGohjpl qS0A== X-Forwarded-Encrypted: i=1; AJvYcCWz6lvXpIe31WIECi3j5S1clWvebyHKMWa6wf1foE18roxqiDef8sWXJTKUtiDfcKcBI1ceqMWoTRXwuwx7v4o/E1w= X-Gm-Message-State: AOJu0YwAgRj488OKzerWLyphR/hHGl8hz6Gc6R4FL0a/PWlOsyUz4AiW uxB/U5oFGTkhb9v9XLjEAKI6zU5uZaXXWoHJlXYpZ9lUJEGto+pb X-Google-Smtp-Source: AGHT+IGUylP38UecAp4ow7R28ofXyse8WlZ1y+8tflbwIT3kfaBZJA8cTz9JYhcDnvYjYa19Kqiapg== X-Received: by 2002:a05:6a00:4390:b0:6ea:d794:ccee with SMTP id d2e1a72fcca58-6f8f38010e9mr17376300b3a.17.1716971346524; Wed, 29 May 2024 01:29:06 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-702220d948bsm174916b3a.131.2024.05.29.01.28.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 May 2024 01:29:05 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v5 3/6] mm: introduce pte_move_swp_offset() helper which can move offset bidirectionally Date: Wed, 29 May 2024 20:28:21 +1200 Message-Id: <20240529082824.150954-4-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240529082824.150954-1-21cnbao@gmail.com> References: <20240529082824.150954-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Stat-Signature: 37a8p5k5nkd31gypsywix4qececx4qkm X-Rspamd-Queue-Id: AAFD440003 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1716971347-875045 X-HE-Meta: U2FsdGVkX1/dO6npdj9r2Jqa/fjq1DYWQ7cAJhsV2MWPCwhT4sYsfyVHPk3XC3Xnf7kQ9Sav2PqtRLfhe9rrHX8h/t2EXMi3/gjsMm1laBkmR7TyzSzvDvU2893YMRcym+IogCcvfX/x1k/o2d6InT1ZY6mFnMtXahpmfMvUjp8nvtp2511aLay2AwTYbqm1dbLIHZTaMY4H3veHyaVivbQCyraY0MsuE7pBtZJnl/u/fAAv91VQLZVrVhwfU6ycuYyMW0SmYVK4twLz2l/RoCfH0Ti2Kdde3OHlH+07zD5k0JrroYNmdzJ6kQhhmheTKDycW9/XzzPsivf0Hoic1GE2PfAP0gyybVkVvLHRO48ikxChcMUXDyFUHxIGnZ4nvqafdDCPoYpNuNm694RxmNYrKIB3dqJ7XPI0PuyiscWHybWdTLNbu4eJFnuOmzPXwYmETo6cc507ixeI5YSf+rvihAahy8D52ajC0xtzhatTXB1e0Fv50BhDr9h51fOMwadkGXVHYVkEH4dtmTmXLpc9IEyCwniRmdST0dxCGFAQTkGtRBA5Rx7JxcQw5aXnGfI15fKjUfGGjfN3qwLXcyWu31ERi/mRWl9ODQTLBzrfPZRdIir7Vq8ylb0CavvXPMz5F6L9c5akHYpnnhJItFgq0y3QL6UdpQQ9MEfKN/42YsVYMrk2SG4CDC0AKKGIDfcR9Un9F8Td2esw095pN71T18pD4BK6pyGk55yU4ujELlSZRmlng8GB6B6thIA0bosuteTyBuJmCSSwacD6sWDfuw8KoRGf0cwCMSkZu7diL+CXVrhYMWi/rcJty9AM9lJO3/bVgDH/bbQBUIe3QJcxBHIqgIGirqUT9eq+wa8Lzt/ZvGgnzffMeTLEdOsvEsayHOkJFF4RLxQE5MMyqYFrKlx555GwmDNiXgOpejccvqcZ7CVr93toDQ8/pKxzMSMtx2imRPL5d3tnaCl PDDMTG3e tFRosFOSfovU2CiTCe5vd2VcD1MxMVJDzc4eHj+Tkj2sxydWnlhWeKprmVMsnKPnem7ww1gV7rV+3HHJ3ZeWM3D7FU/wuULAHYT6Ii2HZd3DooDs8KogKRPCciw9qu3WPOTTa1zEq5hVSj3f9lTmvcKQvs1LguynYVi8aOMUCR/XcwdVnEuFq/EKAeHmhPs9hAOknP/vsuOeOq4mnq2FQmM9iNfJG75NrueHkx1uWh6NxWvTavyMi3ee30nYBzc1B0/6CoXzLyv920m8BwVqUizLZY+hJEWDDKfxV+IiGdw48odX9e5GN8KRU5pPhKWIhEJ3wjOwApIcF3bfRQ/Bt2ftvc/vstuAGDHd0tsgmZ+XvDocq4NodSRVnCa6xzctY0rMA7FnclmtBUw5ZcxyqMJSjhzI93IQ96qG7gd7lRtSI9dS2ltiV1blWx1S+RF1SPP85DwU5TvJ5iYhVomT5hXa2I7vsRy/ps2TYPBCg6t3vnP3bJKaKDOor+SJTLcBsOMXNbx6Yi7auehalMesq8t8lrI7GUKqUcPTkLjLNajLjusvOOUZo58E5czPlMC+0d0PrAtUt5rXBxxZVh7zRDfd1onCLn3jG7ZYfeHYA1+CjVZFfJu33Z9HEJYopBUvyCjQdQAwXcO2wXFU76jmT+CyGAg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song There could arise a necessity to obtain the first pte_t from a swap pte_t located in the middle. For instance, this may occur within the context of do_swap_page(), where a page fault can potentially occur in any PTE of a large folio. To address this, the following patch introduces pte_move_swp_offset(), a function capable of bidirectional movement by a specified delta argument. Consequently, pte_next_swp_offset() will directly invoke it with delta = 1. Suggested-by: "Huang, Ying" Signed-off-by: Barry Song Reviewed-by: Ryan Roberts Reviewed-by: "Huang, Ying" --- mm/internal.h | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index bbec99cc9d9d..3419c329b3bc 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -211,18 +211,21 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, } /** - * pte_next_swp_offset - Increment the swap entry offset field of a swap pte. + * pte_move_swp_offset - Move the swap entry offset field of a swap pte + * forward or backward by delta * @pte: The initial pte state; is_swap_pte(pte) must be true and * non_swap_entry() must be false. + * @delta: The direction and the offset we are moving; forward if delta + * is positive; backward if delta is negative * - * Increments the swap offset, while maintaining all other fields, including + * Moves the swap offset, while maintaining all other fields, including * swap type, and any swp pte bits. The resulting pte is returned. */ -static inline pte_t pte_next_swp_offset(pte_t pte) +static inline pte_t pte_move_swp_offset(pte_t pte, long delta) { swp_entry_t entry = pte_to_swp_entry(pte); pte_t new = __swp_entry_to_pte(__swp_entry(swp_type(entry), - (swp_offset(entry) + 1))); + (swp_offset(entry) + delta))); if (pte_swp_soft_dirty(pte)) new = pte_swp_mksoft_dirty(new); @@ -234,6 +237,20 @@ static inline pte_t pte_next_swp_offset(pte_t pte) return new; } + +/** + * pte_next_swp_offset - Increment the swap entry offset field of a swap pte. + * @pte: The initial pte state; is_swap_pte(pte) must be true and + * non_swap_entry() must be false. + * + * Increments the swap offset, while maintaining all other fields, including + * swap type, and any swp pte bits. The resulting pte is returned. + */ +static inline pte_t pte_next_swp_offset(pte_t pte) +{ + return pte_move_swp_offset(pte, 1); +} + /** * swap_pte_batch - detect a PTE batch for a set of contiguous swap entries * @start_ptep: Page table pointer for the first entry. From patchwork Wed May 29 08:28:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13678365 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB878C25B75 for ; Wed, 29 May 2024 08:29:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6D5C16B00A3; Wed, 29 May 2024 04:29:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 611026B00A4; Wed, 29 May 2024 04:29:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 43B586B00A6; Wed, 29 May 2024 04:29:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2194B6B00A4 for ; Wed, 29 May 2024 04:29:18 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B8454C0969 for ; Wed, 29 May 2024 08:29:17 +0000 (UTC) X-FDA: 82170758754.27.631DC94 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by imf08.hostedemail.com (Postfix) with ESMTP id DD3DF160003 for ; Wed, 29 May 2024 08:29:15 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=acVORvOX; spf=pass (imf08.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1716971355; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SEiK5CEmIDFaJlnWMqRQRlmiXqqRnfHwoYyNloeZy6o=; b=wFxMJ+yl490Mfrg/oyoKhoMoknr2Fc3E0INGCML21DSAaQPnhHIEgtyNTcOWI/urhpEviV LRZQRU1IrU6xN5sAdY0X6U8e/UMYcEE59WWme43jjgDshYTg2TKZaCsHAjcJCmHdKuV6tW z2iT4OEVPGVezT6Oo5/8nQaXoo37ATc= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=acVORvOX; spf=pass (imf08.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1716971355; a=rsa-sha256; cv=none; b=vPl3LyV+vGRrgYBvzp/CN9UyfrduPPq6LwyOmMmhQhWpnYlGZHuDPJJVeHU513H9/IWCvw yJhDF9iXv38ircjOdWpsbASY8N8uHoF5JOERekCYYdrzJy1HL7FB23GT7M9iralvulqrvX F9Lkl3Rl1JlktmeX8S9wDcjsEieEVZ8= Received: by mail-pf1-f169.google.com with SMTP id d2e1a72fcca58-6f6a045d476so1417160b3a.1 for ; Wed, 29 May 2024 01:29:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716971355; x=1717576155; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SEiK5CEmIDFaJlnWMqRQRlmiXqqRnfHwoYyNloeZy6o=; b=acVORvOXb691NgUMAjUpyDRSjrTf0Qsu7IgaKhuSiEQl/SSNM0/ztpcbgsnysGihu3 0x0QNp0dLID+AbUTBIGi7u69qqenFHbMvt/Lzh/r2CeYxMevE8sXu8KQYqmxO4xhCS+R KW6H5eAC1n8rfg65v3e1f1v5z3VYbZ76ArvwTadsmfp9/W3FLdyAwW46Cbtrpu9NqGQa EYQHT9RRJ/pQxFBpVPTtynjkLW3Qk7ygzD1IHDVSJkddufSaxBdd2Ge4iyMlkInJiRs5 /ryXwSiZHmQ2YicqIC156h3VuN9DFe1d1y8N3o3DbhFlKlOdpODBw6c53VXsgbX/sP8A DKsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716971355; x=1717576155; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SEiK5CEmIDFaJlnWMqRQRlmiXqqRnfHwoYyNloeZy6o=; b=RFzB2FAcLxZLVxEqTwYEzwfH0p01EuC4G0bba2gHhLk58LmtmVSePqoYk7Vnqxbt6r iLRPjzNuH7k/DOnqGGdoxo1rxvxqsfFdL3DGwwrp4UbV5Wn0Jp/b9/+HMKqGrCbkgPee HAueyuQHGJwzIPG9tyE6TZqKRlwqAtsYyLOB2x7xTx05e2LXMHSAZllQTlC3MzDJ6f00 nC1fNBczl3n2libW6UukKz7RrCmx1+RSBoW3HVXoizEPjFLy48iL9vn6jDuCGfaaB9qX ovA++INp8IhnWwJKjrPDShQUnZAXgscxXqWAUas/XI5urlCU0AbfurA1U6dEie8doZ0w 8e+w== X-Forwarded-Encrypted: i=1; AJvYcCU6Dl7SA1YCcjRzupsOetw90p/zaidC/cmaklVzsClEVd3K/rb2vXpNhCOtK3gvMkBodIYJ2zsUms72q9sYeXw1U8Q= X-Gm-Message-State: AOJu0YxFfs/pTy9jYZyYTlONMvXTPx3k/R6jRf6MnxXqx3yqIyp10Qsy I+Rl9eIP0Ysv5/BcLn8rnIu90AoUgdPzHUEnDOm15MFHPLZJcDbg X-Google-Smtp-Source: AGHT+IHC+4YVrv2f2RKD1f5/1XnNvm559LObbEuRJtkuQpqFiFl9jlcgtsUPga8jgyaa5WCSFkT/bQ== X-Received: by 2002:a05:6a00:908f:b0:6f3:ebc4:4407 with SMTP id d2e1a72fcca58-6f8f2b49544mr16440786b3a.4.1716971354683; Wed, 29 May 2024 01:29:14 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-702220d948bsm174916b3a.131.2024.05.29.01.29.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 May 2024 01:29:13 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com, Khalid Aziz , "David S. Miller" , Andreas Larsson Subject: [PATCH v5 4/6] mm: introduce arch_do_swap_page_nr() which allows restore metadata for nr pages Date: Wed, 29 May 2024 20:28:22 +1200 Message-Id: <20240529082824.150954-5-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240529082824.150954-1-21cnbao@gmail.com> References: <20240529082824.150954-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Stat-Signature: dthkihh9hyewgn94pesxtedmjbi39sx4 X-Rspamd-Queue-Id: DD3DF160003 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1716971355-269458 X-HE-Meta: U2FsdGVkX1+mRHrl32lMCBWxjNik/oXLl7ihEDXT6hwDN6CFALkQwUUsmhjPF9W/v998yvstuzLIAug/NtDTafCHYLv/1OmkeZ+lU/eLRhf1RpH++CQ6Gu5ksDOraMlnu5QMTy1aWVV8EPzmMXdF4yhauZo/6+6uzWju6OBqBeyA5joH6hAuzaCN46B7LcgCWlCuSrY3OcqtjdZn2lyiMMNtdo8GRWg2YPe/STndo9yI/2n1EaRrpVFnC52/np3loRvBaZ8n0uK5bT1m9Ef/DNmIDrudstkCCSTek1wwH3iGQasSOu9gvos379GhEA8PCi4X6haYjVQ9erztdz1WtgMtmkhP+ikSGrlIl2b29eQUkuZ1oMnS2YhsaVSZwHnADPjVmhpGYwlNTOxHzqA8HdZSDuhU34Fdj8JOSR6vXYf37qAJX2Jq4z8jXJDtXGSsW9cqdZBHUPOn4FsQykNGMIgSL1W/2mZHhlUn4cudYHRmO1I43gjasgV+tZcd5RyZn7Ts0f63dVwK5Fegs4/hvQDU9KNFRgM+o4oTEVCfl7q25TAulOpH2NLWuwrmJBeizRZws+DOxU4UJV07GOS537fZsHhQ0g4iOXuJqyLSnneeo7lmfKwZ49kJp+hiZRkHDUpXyYKBy4CA2U9hVCuCTGXWe8/f/8bBA0c3hxD9V/UJaUA8YjK4jQrC7y/jxdRyUuex99s4NMyRSIVHCD3XFRVMh/EqfUCdWIWTJ8NGeKxsftl3tMci29LcJdvnS/vceRPl8RMYzz53ZkX7UFYkoUPOUN6LhpoQ3PSx4H/FDIcuzab1p/lfZAJWETBvd+uQDFOKryUDVnSVKzIlY0MPloKgF+lA9/wunG6ArT4xXaUAnFEMDDS2VgdyAo5/3HqE+8cHskdvZblHf+t68MXmhQAYklu4/vTa0L/O8A01AhS1aZ/O4I55heSv86CSNKil0dxFTfmynixtStGfFe5 980Sy5+o 7wnDc7Xn/cGuRxP3CdNCT6xzaUfxISzizv4sOpyzKgdWK0vf8x9hpX5/vcAskc73LuhORjCMv4e88uUkdKkYWD47FSghYSGsTlKhkALoylBPRSfWP1lYuUNi4YAockquxu6TMw2LxabsYwTROd5w7HMisY4eMFDrqWuc9J6GfDfButcW+8ks7K+V+6GYNMhpQ4x6HVFcocb6rSpYzqT7wwszNIkOy/WxIg7yLMmXdn04uBYyEwe/L8by7HcRv/FVXOczDhbw3YhDBWGb5mxQSHN8I8TJ3ncmvRfal3n3ensPjnnMXSkYTm763q9rPWYdWXdAdwYR6VJ83yV7WHvLu5O0Qis4PIoHfdLt0soVAWGCUubGYBoSZstShyYoi4zXCImKQMQ8fumKkFoRqmzQ2gbOsnKcZ8f4LgKsg68Lt4C7exomhFMlncy20UzbhhoAxOXSqvxxyq2jV7JBuVonuWXFwx3tKmtHgZuJfPUF5Lf/yKDZH+OUTF+nvAeRAGWRi8IGHYCTkWPATTznJ2HG1bOd5CtCa5pV9KmExglNpienHoGVxASJLq/gbeZQurNQZPlLkOZz9+r3LoZXn3EWZvDGaxzdOjj3EjsVeScUfjUV7KX1DeoZAOB5LYO0AR8iBkZpP6+njS7luOrYvaecWyq4mrCi0BrGFQWXkyrhSQa9EX2Vv5ItYyTd080Mz86lAf5pYLwRDie2Vtan0FA3KcyoKg6rjKdSeHt3ZcYuPULdbHKpBfnKWdVqWIPuoEN9qymjPIxmdHwV1+erTy/9Pi+tYhw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song Should do_swap_page() have the capability to directly map a large folio, metadata restoration becomes necessary for a specified number of pages denoted as nr. It's important to highlight that metadata restoration is solely required by the SPARC platform, which, however, does not enable THP_SWAP. Consequently, in the present kernel configuration, there exists no practical scenario where users necessitate the restoration of nr metadata. Platforms implementing THP_SWAP might invoke this function with nr values exceeding 1, subsequent to do_swap_page() successfully mapping an entire large folio. Nonetheless, their arch_do_swap_page_nr() functions remain empty. Signed-off-by: Barry Song Reviewed-by: Ryan Roberts Reviewed-by: Khalid Aziz Cc: "David S. Miller" Cc: Andreas Larsson --- include/linux/pgtable.h | 26 ++++++++++++++++++++------ mm/memory.c | 3 ++- 2 files changed, 22 insertions(+), 7 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 117b807e3f89..2f32eaccf0b9 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1089,6 +1089,15 @@ static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b) }) #ifndef __HAVE_ARCH_DO_SWAP_PAGE +static inline void arch_do_swap_page_nr(struct mm_struct *mm, + struct vm_area_struct *vma, + unsigned long addr, + pte_t pte, pte_t oldpte, + int nr) +{ + +} +#else /* * Some architectures support metadata associated with a page. When a * page is being swapped out, this metadata must be saved so it can be @@ -1097,12 +1106,17 @@ static inline int pgd_same(pgd_t pgd_a, pgd_t pgd_b) * page as metadata for the page. arch_do_swap_page() can restore this * metadata when a page is swapped back in. */ -static inline void arch_do_swap_page(struct mm_struct *mm, - struct vm_area_struct *vma, - unsigned long addr, - pte_t pte, pte_t oldpte) -{ - +static inline void arch_do_swap_page_nr(struct mm_struct *mm, + struct vm_area_struct *vma, + unsigned long addr, + pte_t pte, pte_t oldpte, + int nr) +{ + for (int i = 0; i < nr; i++) { + arch_do_swap_page(vma->vm_mm, vma, addr + i * PAGE_SIZE, + pte_advance_pfn(pte, i), + pte_advance_pfn(oldpte, i)); + } } #endif diff --git a/mm/memory.c b/mm/memory.c index 100f54fc9e6c..c0c6de31a313 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4314,7 +4314,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) VM_BUG_ON(!folio_test_anon(folio) || (pte_write(pte) && !PageAnonExclusive(page))); set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); - arch_do_swap_page(vma->vm_mm, vma, vmf->address, pte, vmf->orig_pte); + arch_do_swap_page_nr(vma->vm_mm, vma, vmf->address, + pte, vmf->orig_pte, 1); folio_unlock(folio); if (folio != swapcache && swapcache) { From patchwork Wed May 29 08:28:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13678366 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 360FAC41513 for ; Wed, 29 May 2024 08:29:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A325F6B00A7; Wed, 29 May 2024 04:29:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9B9706B00A9; Wed, 29 May 2024 04:29:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7E4DA6B00AA; Wed, 29 May 2024 04:29:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5B6936B00A7 for ; Wed, 29 May 2024 04:29:25 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id CDD08A180C for ; Wed, 29 May 2024 08:29:24 +0000 (UTC) X-FDA: 82170759048.18.5400DA1 Received: from mail-ot1-f49.google.com (mail-ot1-f49.google.com [209.85.210.49]) by imf29.hostedemail.com (Postfix) with ESMTP id 1012C120003 for ; Wed, 29 May 2024 08:29:22 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=e4GomEHM; spf=pass (imf29.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.49 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1716971363; a=rsa-sha256; cv=none; b=4KeNbIgaEITNC4UEkVC/coQy4UdDlkuO6HXd65ootCeKvvMxxYbmGsRCSPTbFgg/PeADm1 nD0ljk+QwMuVS5HcmCqPsiPnb81EuBZ0Nv5aNHODXuen9cG8rsqPheQH2++P2R7uyuODx1 DS0SzqW2X91y+KJdlRxj/Ucr3yBfOfg= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=e4GomEHM; spf=pass (imf29.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.49 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1716971363; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6+4WZx2jk6Uy8Ser1SPRLQ9xpkk5S7WupCS1+ZFnTlU=; b=OPaEGWBqIijyJaY+02PpVnr3jNWgApjRQMrlxNMdp970Gr7Min5CWbVgNV5M/9UATozbeC QCql8fjs2gIxAdE01orxnXoJ94nq/X74AK8GbdXmLf1nspdF6W8VyfCMecB1Xg8dGHZWX2 Yo/hPa2ptXQrj36NcS6bOaRrwJ+vkC8= Received: by mail-ot1-f49.google.com with SMTP id 46e09a7af769-6f8ef63714cso1115385a34.1 for ; Wed, 29 May 2024 01:29:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716971362; x=1717576162; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6+4WZx2jk6Uy8Ser1SPRLQ9xpkk5S7WupCS1+ZFnTlU=; b=e4GomEHMSfoM5GCvxrCsOj8frjXebO0h1IeIFQGQqXjPB++sNpJCcQUk7bAtNTHAGO jrGAe6xKDyi/91Z5M+yaG/GxMCiA2Et6Bcxt5G/QQaRUBc/2V0480x/CFDcqBZpQz3SP 79YMjJONdH82o7Vr18+RmeZSyts1CPG94vuKffWHkmZ+OqndicjMPMNEL1jK3xFTsYNs odTyOlE6yenHMIyeT+gLrIV9+Q3HKDPnOsr3L3oO9J4oLQwm7zcNX1KzFpe3T6nOqSge OBq3LBgb+1LJXI3Gkqw1Lqg16Yz7meZK8aNpLGTpELaYzfsvmhbGELAzMEofcZLui8+1 stAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716971362; x=1717576162; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6+4WZx2jk6Uy8Ser1SPRLQ9xpkk5S7WupCS1+ZFnTlU=; b=NLp0gNLhnG8t+hSxrxsc0BH2an7YE6ZWKfl37YU+bFbBnldINBie+F7rknPL7YdPMA UKQfzD/PBauQllv67JgAljRAGrlepmfHhZK9sGgyy6ldccGkSCcIzDyQTw40Sf47fmwE 7aGQ5wGGBrCDooaBBbsG4rf+EdrooxbwEAyz/BJLenvJTn7kxguO4eptcGCxBn7mLtLd u87YnwZr9ZAoJ5feH8rT92igqsT8XWJAxFTfTssovDkewGifJhAz4RP+1MENJE2Ij7qk yd4iLOloRS9GWyBNc/zyRvkQMmeFZ7lHlg9HJpitZBmOqxjlRDDpm6RCn1uj+sYJ/B26 WdcA== X-Forwarded-Encrypted: i=1; AJvYcCWTHK/0bY6AcJkwJAAWORnocFqwHH00gf20dOorc3Ruj3YQBIxiLF0A1fmZ2Zf5FhBrKDAg5sPSzY9uD5tcylUp1T4= X-Gm-Message-State: AOJu0YyHr/4GszqYikxYMXtJBvsW9XKEZZ+nkpef1HkK448GSb/XDBu9 L039GD8+tkxUwSkGGWUpXLvplk22onyzjt+3ZsigOLJelmbeMVOh X-Google-Smtp-Source: AGHT+IHGb+sXIn68+MgGWVaSAFwEZ1WhY2usjFaXYWxY+vdoWKkWciGEogVMz08VfHXPwQpQdXzQOQ== X-Received: by 2002:a05:6870:d6a2:b0:24f:e000:a6f8 with SMTP id 586e51a60fabf-24fe000ac34mr13014665fac.41.1716971361783; Wed, 29 May 2024 01:29:21 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-702220d948bsm174916b3a.131.2024.05.29.01.29.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 May 2024 01:29:21 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v5 5/6] mm: swap: make should_try_to_free_swap() support large-folio Date: Wed, 29 May 2024 20:28:23 +1200 Message-Id: <20240529082824.150954-6-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240529082824.150954-1-21cnbao@gmail.com> References: <20240529082824.150954-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 1012C120003 X-Stat-Signature: uutkwxws1urczwfgs6hrbkrqtj1ebi5u X-HE-Tag: 1716971362-190664 X-HE-Meta: U2FsdGVkX1/0vMvHBE/BTWph0zksCP2rMJv0xSQbJrHLSlCvCEx7zSXGHJDrXjXjsnAnaPtGIfH/LSqY27Q4WAoDs7gr2EBTL5HO4PGvqvzj34jWmL6hM5OBkJOGT3tq/ws1aA1l7SYogJ5/h6uy3DL6rtRAEnH3flnwbuJOu1qRq2ml5J0dKLSydDqigg18VueOlEP1lHz7yzcmHtCcKJZ/kc/TwaXoblcyrb7kL2cmAVnXi44J/wCt6zO6jXz7nWpzHCtIBCyXSV/Y9QOKn3Dj31fbUsn1RFOcYQkrwJCAAHq+yYCyV5y9h00RcV8KHdsW/eaTEzPWkJNhEkYkEqHHs18NMHCmgTJ7AdYqHV5d7HxlSkF2JJ8hkRnHzK5h0X/HNQdQulUKWKmR1h2M/PcJm8SMeCjkVtIeQ8TunepDoS8RPT+4G+elRrChRfHEYqJAbHFe1s9pR6vP0DOOG/bkuiT9EzpdQdp9KNiwwEfxKb1hhlp1y3U2z/xR0tJzutMENNZ1+2IpZz9ZUkcCap6SIPzjiOA+XrB/D1OdkjgX7ywXKE6uiDC/RDTQLMYiMryR/d5vpjED0ZeyY+S6KpOgdNFDf9ZKUwlsGakPA2zRxYJtwH5O2ub8XJMaJVfB3om/TgVkj4o/fvCWpCyWdDLI2aHoBMAMJfegOz2CnacKMMGULaUFXp5BuTIfkMnxjt0Q9D6Hf59zU9JOUbSB2aBEyOm1SI4DzxjbYsZF7mKRpuU42/QBjwwtbRsAOgqMSCoWdeKGEuwbOkplK3hGd1tU8QMKQSCNCErCiAQAoquABJaTGMuSLZVa8Yi4TqsFe3yCNr9a5Yi6Tp+5wCmh3okDMIUI+vGnPf2FIREDRn0THsvB9BiwdYp61UTfqfWskfv6XuwNyRUGxdxoGCPPqkkULPm/9WcuW0bpvQO7Cpw1eGijeClhG8e6p5EHsenTeMokc2ST6qGXMA+hFLH HdWqW9jE SLhhZJyK1HPlLYtC4TfUOqz2Cd3K8G6Itd3Ofrozp+d3/ZiDZ0AwtxtGxSM+GREaslauTscWmf4JwBDAK1H9TEpiSnvQjMjNKzee5sKaO1/qK1foR+bmegPqbRJSTEzWg4StEmt9vLC5xqf/vmB9uAelylnh0BVlv3uqTt7Z0e4cuTyR1dLmSbTu+sVvfAl6UEwkzuuFvLq+1yxzIl4dM0vbI/EYy0tDQfr24hGbpAoa+b696sJBMo9/yLi/U71kkb7k1tuc5N2kEWm68I00x3IVdAhl8Nr2qulEt//c5ihfN3CVl4VvPuOi2ZwViJqUDLEPt0hPTHVwuJ0nbbaLrTjSPDe7FgsUhTrt/Uq3KB4NGVeBB7M2h5xXipH5WX3aKjmqUrTZGxWAFq4GgQtGHdzCJUD+0gWSzXM2mdQ5GF0Eu0+1zHMTKtbWTF41vYdG4H2LaigQ53pMaJEhVSNaRnyMsrvudv75hkz2fLQUIwAcrpxXrkJ7/lD03CRjGCzBW9MrzvCCuQFPamBKIIoSvhma3jkLYdvNlk7LLzE4XcafLQfoufjatoSv4da8kqVl7yJ0Xve4w1T2GQIz/iCSwxZj4X0l1h62btNsmViDtb2eLfdoo3793226PehExCrR3EpTLoi8L/YtMU9jSjgft5yC73BAo88bXgxIadlf8EIzipIqjNrHukVU794c45myAd3b61dQGu8TlrYpI7qY6q14JywRfW7/D7BsrbTLTTPoECXg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han The function should_try_to_free_swap() operates under the assumption that swap-in always occurs at the normal page granularity, i.e., folio_nr_pages() = 1. However, in reality, for large folios, add_to_swap_cache() will invoke folio_ref_add(folio, nr). To accommodate large folio swap-in, this patch eliminates this assumption. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Acked-by: Chris Li Reviewed-by: Ryan Roberts Reviewed-by: "Huang, Ying" Reviewed-by: David Hildenbrand --- mm/memory.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memory.c b/mm/memory.c index c0c6de31a313..10719e5afecb 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3925,7 +3925,7 @@ static inline bool should_try_to_free_swap(struct folio *folio, * reference only in case it's likely that we'll be the exlusive user. */ return (fault_flags & FAULT_FLAG_WRITE) && !folio_test_ksm(folio) && - folio_ref_count(folio) == 2; + folio_ref_count(folio) == (1 + folio_nr_pages(folio)); } static vm_fault_t pte_marker_clear(struct vm_fault *vmf) From patchwork Wed May 29 08:28:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13678367 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4350C25B7E for ; Wed, 29 May 2024 08:29:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 59D596B00AA; Wed, 29 May 2024 04:29:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 527666B00AB; Wed, 29 May 2024 04:29:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C6EA6B00AC; Wed, 29 May 2024 04:29:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1A0B76B00AA for ; Wed, 29 May 2024 04:29:32 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id BC550C0971 for ; Wed, 29 May 2024 08:29:31 +0000 (UTC) X-FDA: 82170759342.11.90E2B98 Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by imf26.hostedemail.com (Postfix) with ESMTP id E9DB5140008 for ; Wed, 29 May 2024 08:29:29 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=fcImPoCn; spf=pass (imf26.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1716971370; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kxE2mAKSIadz3iJIa+4kx5y9ZNqImGpGMb9QcvFrG9A=; b=tlUc9+WerhpSFvP1M+LO8rmPYiNPSbtwE7YEaVvzVLm0B2QG/Xo8jyfsiFeThSp9pxnsBC zCxyyFjPe8f9gS3BYvYCAfZfTwpyYpj8ReVnWwEJCuHovr4RxgqWyfBzYJ8PSx+q0Lt0Bp pSxZDeuwwXjfhLV4NI/pJJVuLTHMaUI= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=fcImPoCn; spf=pass (imf26.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1716971370; a=rsa-sha256; cv=none; b=VlfsGPCPDlo2ZrVXmPSIUJqTkv5MjVy0cT7ECuMEAF35pdWAqPrPmqhNeryiO7rUjCjt9K aGiWTa8ncW4+a5z9RppCR+AdtiaPRdGJt5O9+C7OMSPZCMRt4AiHCLx3VJJSiJOa5ISMvX yaJRb7za5HzDxuYwxNY5UvfOqhhUlbk= Received: by mail-pf1-f177.google.com with SMTP id d2e1a72fcca58-6f4ed9dc7beso482539b3a.1 for ; Wed, 29 May 2024 01:29:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1716971369; x=1717576169; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kxE2mAKSIadz3iJIa+4kx5y9ZNqImGpGMb9QcvFrG9A=; b=fcImPoCnybjRJ1JovokCu6rmF8kAbylDUZkWnodSIGF2mNE7JVHJReYeiUQlKxo6xg Apz58z4bB0C7jICaFyi5PWR7Go8aZH+edeek1tiXDrRu+hrubZIlAruP/opGZI6cH7t4 VTnCwXSuC8CVGpCNH60yfgi4N8K4O8lIf5oq9E1c+JTgJA87J2mmGo2fYCONA0YzWuqI 7tS5vNsCktAoKELVhT9b3H3KL0fflVvd/RdaJqPNQJG2myyh1RmU+aGLpkOSfoCANSSA PBCLGmJ5AxGNmvrgJnV3wmsHtnqJNKw9pslZWBqwcnqnvMrSm/pB9N3iK7Vf5vk/GTfs S16w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1716971369; x=1717576169; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kxE2mAKSIadz3iJIa+4kx5y9ZNqImGpGMb9QcvFrG9A=; b=q9z65IODpgykV4yfjAEC62MTExlCQfNtNfbpthRObclPGf5clfvOlmuaielISyQvw9 jrb5cNFFwkjtx9Aaa3nkp2M/LljtXLqiNuzPmXbCrXnvKUnuXtIwzgvQq+RYmO3+K33p 1PTDL6+OHFdkSwUl11KD1Iwsu9IgF1Aj+zEDKZ/umKfYd7W1rQ65UCPA48e5tRCb1OJh Yp9ZKlHix6mkNquKVHq4wH5qP0WsNIuO73IcIO08SZIl3zha3dt8mHLjmnbHFClQVN7y /e0pqHZ5rNwrOeEvSb5nuCYpA9UftQrJPIU1z0GQAVZEu22VMcWHTPBPmRNO46xWfNrU UNJA== X-Forwarded-Encrypted: i=1; AJvYcCV7D/166oPE3BAOdfYf1IuXmQ5zEPJA6A6CGAv+gxRjC4QNdp+s2IHU7PyAY6MEwJLk6EEUcUlWgQs9i6KVtjVS3iA= X-Gm-Message-State: AOJu0YzQnbGMAGzO3JO09oPAu5WXTgKp91VnyE3wzQ1u4EyWYXfUFGHQ qYRhJHWpcb2QpvQHqaRz0/1bDmeRqk89UxwJHUPmgG7kIo9fUSWz X-Google-Smtp-Source: AGHT+IEbSGge22ETQ01VYggngJlR3UpTZl4VLPzOebNpH6caOUJ2L4BysvKHtwvqOmVKwJG58Oyn5g== X-Received: by 2002:a05:6a00:2fd1:b0:702:1d51:98f7 with SMTP id d2e1a72fcca58-7021d519c44mr490666b3a.2.1716971368774; Wed, 29 May 2024 01:29:28 -0700 (PDT) Received: from localhost.localdomain ([2407:7000:8942:5500:aaa1:59ff:fe57:eb97]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-702220d948bsm174916b3a.131.2024.05.29.01.29.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 May 2024 01:29:28 -0700 (PDT) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: baolin.wang@linux.alibaba.com, chrisl@kernel.org, david@redhat.com, hanchuanhua@oppo.com, hannes@cmpxchg.org, hughd@google.com, kasong@tencent.com, linux-kernel@vger.kernel.org, ryan.roberts@arm.com, surenb@google.com, v-songbaohua@oppo.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yosryahmed@google.com, yuzhao@google.com, ziy@nvidia.com Subject: [PATCH v5 6/6] mm: swap: entirely map large folios found in swapcache Date: Wed, 29 May 2024 20:28:24 +1200 Message-Id: <20240529082824.150954-7-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240529082824.150954-1-21cnbao@gmail.com> References: <20240529082824.150954-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Stat-Signature: ji6dj6ityiwgbm6r945z46n4c6955ndm X-Rspamd-Queue-Id: E9DB5140008 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1716971369-374697 X-HE-Meta: U2FsdGVkX18uz2vde7Ff6AFWZrNO+6d1JOtX2CXD7R4H4QNuBLovPHfxNFvQYBRASCoCkKcpBSUeHy7YYW8bszJhUnWRgdzi4yCIG2uMlwqIEp3+BR2DmXH4F4uwgMIy/RQ2iVje7balOYFmEdDH9eU4Zi0DwAMdobSTiJQri+lkS3ARCjtEGkD50o78LEEXsyNmStvFCDKc9FiLSkWNCNoGkx3hvHGWjuk0pjIbETaLewkCF0ASq8K3AXLLNdLLWHTdQ/y/cwK0PMoWRdZHfyPqXgSZDuwGmm2yF0aABvr8QyhdAQvekHHB1685G11lfPy2Zg5Ti74y4kDzjUWokcFN36UVzsTiAudonQP6tg3NZy2D8v5bOotGm5IlO3WmdVnXcObmK1XJLuQ7lB17JQuAdKLVjVOE3DMKzxjw1rYNV3fVwZEGnlyNA+fBy87HTvlnTVLV2LdsU9gOIQA/0I8yqGIGr/3cIhcUAZbxU33xC+TrmqjHJ3KLHgQB7rMkb5z+D7pvKuwYvYvlQDDDJ52hfkddXNj34fab1FR9D+DGfkRh9RvPjAv8dxO8NPWKlnaXQhnNa0YQSFnnnU536/fY2sBxIuU2z3belIMPZzB3/JclA+8qDPbOQQVK/ic9mOoeqW1CLASR4n/lfTtthqJKQjJXmEdfk56uVY/eb492BF/FU7/FOkAYRGCGn6qNOmAs2RKaCjItcB6ogtG+AbUzPidzkvgZaCq1CGLkc5bZOOpVlTkh5n/xT4rlXuoB4Yx7u/fOiWmJf8XnFpvZ5WXgQGwsQAwctNO/YTWbLZvrgWXzy5/QQXXCt8+11/sJPHgFOFfHRwgPhNFPJs8x1SkzkzTnGQ793Clvp4+FVUFhdkJj1Q9PyDdNy8o7cIL6dYX2DS3WMIRMA48PJ2oLTsxYJnimje+hhLqzkHTUXRF5XHvA3Uz7Cr1/UTTaQ/fYcywBABkUhVj8h+gs4La 8AWYT3y3 NgammXo/PffFhZw/juM2j7bDGKO0fjarfDOc33+Rje8mptCviINWLm/u3+s/DT9cN2gRA2hbZXD+5xYv8KRO2PLC5p9hoilpyAX0A1IDhwCPBxf2PTkV+Jh7IidfVgLWRXT1ioM2aeoI2inexmfZfx3kUuL8T7JpkWDiWPCerWiZaC5tZ4IDVMgQpA5dAYxtXLT/A7RCS/x3+WMyiUHCvqoZpaC54++pG9BwjfAinicnNVFe+vXYTtguari2j1D0M1aLp3LZSneSVezevgEJ4Bb4YwD/M8i8w3OTBF3j4TLST5c7EfmKTuHDwKFGK8qZwcG+d31pCo3W+L4lNesChg6YKwYbbO15ngWbfReLqoqES6qFVf1hszXoz39CVRBJW9g9iCQI9VmPU0v0IAJofzQ5uIICuMeYS4PtvHnwFhxjRNCBozUBku8re+rCRvdsXpn2AlcK/e5ja2gTHqulm3a0vz8ANi8O6nLZWV4rGaCCh7sKv4p1TC0iKJYh2p9gyuvo3nkp5oG8l9woKxbyLeu8pgEn6Gq8JgSHOPizpmux6h8hWZOaGNhcYvBhGSxzKVXz9hyc2H1SkOFDgmFdOt/G86gJIeEzkyC/n X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chuanhua Han When a large folio is found in the swapcache, the current implementation requires calling do_swap_page() nr_pages times, resulting in nr_pages page faults. This patch opts to map the entire large folio at once to minimize page faults. Additionally, redundant checks and early exits for ARM64 MTE restoring are removed. Signed-off-by: Chuanhua Han Co-developed-by: Barry Song Signed-off-by: Barry Song Reviewed-by: Ryan Roberts Reviewed-by: "Huang, Ying" --- mm/memory.c | 59 +++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 48 insertions(+), 11 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 10719e5afecb..eef4e482c0c2 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4016,6 +4016,10 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) pte_t pte; vm_fault_t ret = 0; void *shadow = NULL; + int nr_pages; + unsigned long page_idx; + unsigned long address; + pte_t *ptep; if (!pte_unmap_same(vmf)) goto out; @@ -4214,6 +4218,38 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) goto out_nomap; } + nr_pages = 1; + page_idx = 0; + address = vmf->address; + ptep = vmf->pte; + if (folio_test_large(folio) && folio_test_swapcache(folio)) { + int nr = folio_nr_pages(folio); + unsigned long idx = folio_page_idx(folio, page); + unsigned long folio_start = address - idx * PAGE_SIZE; + unsigned long folio_end = folio_start + nr * PAGE_SIZE; + pte_t *folio_ptep; + pte_t folio_pte; + + if (unlikely(folio_start < max(address & PMD_MASK, vma->vm_start))) + goto check_folio; + if (unlikely(folio_end > pmd_addr_end(address, vma->vm_end))) + goto check_folio; + + folio_ptep = vmf->pte - idx; + folio_pte = ptep_get(folio_ptep); + if (!pte_same(folio_pte, pte_move_swp_offset(vmf->orig_pte, -idx)) || + swap_pte_batch(folio_ptep, nr, folio_pte) != nr) + goto check_folio; + + page_idx = idx; + address = folio_start; + ptep = folio_ptep; + nr_pages = nr; + entry = folio->swap; + page = &folio->page; + } + +check_folio: /* * PG_anon_exclusive reuses PG_mappedtodisk for anon pages. A swap pte * must never point at an anonymous page in the swapcache that is @@ -4273,12 +4309,12 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) * We're already holding a reference on the page but haven't mapped it * yet. */ - swap_free(entry); + swap_free_nr(entry, nr_pages); if (should_try_to_free_swap(folio, vma, vmf->flags)) folio_free_swap(folio); - inc_mm_counter(vma->vm_mm, MM_ANONPAGES); - dec_mm_counter(vma->vm_mm, MM_SWAPENTS); + add_mm_counter(vma->vm_mm, MM_ANONPAGES, nr_pages); + add_mm_counter(vma->vm_mm, MM_SWAPENTS, -nr_pages); pte = mk_pte(page, vma->vm_page_prot); /* @@ -4295,27 +4331,28 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } rmap_flags |= RMAP_EXCLUSIVE; } - flush_icache_page(vma, page); + folio_ref_add(folio, nr_pages - 1); + flush_icache_pages(vma, page, nr_pages); if (pte_swp_soft_dirty(vmf->orig_pte)) pte = pte_mksoft_dirty(pte); if (pte_swp_uffd_wp(vmf->orig_pte)) pte = pte_mkuffd_wp(pte); - vmf->orig_pte = pte; + vmf->orig_pte = pte_advance_pfn(pte, page_idx); /* ksm created a completely new copy */ if (unlikely(folio != swapcache && swapcache)) { - folio_add_new_anon_rmap(folio, vma, vmf->address); + folio_add_new_anon_rmap(folio, vma, address); folio_add_lru_vma(folio, vma); } else { - folio_add_anon_rmap_pte(folio, page, vma, vmf->address, + folio_add_anon_rmap_ptes(folio, page, nr_pages, vma, address, rmap_flags); } VM_BUG_ON(!folio_test_anon(folio) || (pte_write(pte) && !PageAnonExclusive(page))); - set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); - arch_do_swap_page_nr(vma->vm_mm, vma, vmf->address, - pte, vmf->orig_pte, 1); + set_ptes(vma->vm_mm, address, ptep, pte, nr_pages); + arch_do_swap_page_nr(vma->vm_mm, vma, address, + pte, pte, nr_pages); folio_unlock(folio); if (folio != swapcache && swapcache) { @@ -4339,7 +4376,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) } /* No need to invalidate - it was non-present before */ - update_mmu_cache_range(vmf, vma, vmf->address, vmf->pte, 1); + update_mmu_cache_range(vmf, vma, address, ptep, nr_pages); unlock: if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl);