From patchwork Thu Feb 29 00:37:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Barry Song <21cnbao@gmail.com> X-Patchwork-Id: 13576148 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64CF7C5478C for ; Thu, 29 Feb 2024 00:39:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F05EC6B00A5; Wed, 28 Feb 2024 19:39:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EB6686B00A6; Wed, 28 Feb 2024 19:39:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D570C6B00A7; Wed, 28 Feb 2024 19:39:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C3DDD6B00A5 for ; Wed, 28 Feb 2024 19:39:00 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 77FCD1A10D5 for ; Thu, 29 Feb 2024 00:39:00 +0000 (UTC) X-FDA: 81842981640.06.43476F4 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf16.hostedemail.com (Postfix) with ESMTP id A2E41180008 for ; Thu, 29 Feb 2024 00:38:58 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=aiaYCD24; spf=pass (imf16.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709167138; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qNEaXVngxEEVFW2r7SF7/R/SOQEyIxB42KfKsKpMTi0=; b=Cq7WL3+xiULyHe3J+Qjr0iJiYI1m6lyJ2P6RCRWLUIeNkl0O0/vtyj4Bdo9spndsqVoe9K xi0OYqbIrftm2/nV1yAqqBg6blYvw3qn1OMaNSr0siDYaBcYpnbDEPeZlDlNJUKaBlVm9J b9tHJJhmzMPqdYGZHnQZxPhBCxJYriM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709167138; a=rsa-sha256; cv=none; b=dTc3qtxU9fVZI/Y+K+BVsJYsLxCIxcX469cJr3irT871ZMHGY2MgSMCTtdKyKY8t2eCsty J1tCMB6W3P2DJkZI58+HV/kZfG67GhCBvPsRTfYTWDiY7QF46q7C2OnpdWxZHnz8igKmuF qzr9D+054hw8a6EDko225565Uffghfg= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=aiaYCD24; spf=pass (imf16.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.214.177 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-1dbd32cff0bso4131115ad.0 for ; Wed, 28 Feb 2024 16:38:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1709167137; x=1709771937; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qNEaXVngxEEVFW2r7SF7/R/SOQEyIxB42KfKsKpMTi0=; b=aiaYCD24J51yD6WFvR93JkCRE4YdEg6yyjnTYcOqNiDPziQoHE1yJPg1WmM1K6dd6a MY41A2SU8zdPPeXxc6GHAMj7ECi5ncZQ/bk5s0ZPyhhlJcc3nfoxDn82o7HLg8Ko03cA TSAw62CSF6/CP28TBWEhBc7b31DWdr3QdGuKitvjoM8ko4wgI1x7ZaxQBIt+wEGGXSd+ s+HUk9zBbj9tbP9R5dIxZ6eFf7q0dVATESTTDeBaQH9P0Zt4br2V8T+dv1jQXPjCQd2W 8LsGSUyDYiSuiJ2B4QmymuGb4X+QDo2TAbvQsGFn7FyqVVGcwdqcaHrufkG32JQLS0UQ 1Yqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709167137; x=1709771937; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qNEaXVngxEEVFW2r7SF7/R/SOQEyIxB42KfKsKpMTi0=; b=VAL0OHvube2tDcQ+MMuXiGYboATMMulkCDZ+UQQKYnBdLKx8wkmkXdpaUeXYv/pbbz 1Aq9+zmJEnouYW11TslWCtZOJEzbm2NY8+Rra/61gFyytFViP5CWIu2zE7ZwiZ9GQu7J s3WzWG6nvIMAQR1I2QTUtz4x0DLNckx3Oinu6KY3uZNeNFKSPvSprlifuqE/kNHf7I+U HDxjxlvEQBholFNUrFzC+pkmNugKbSW2eBaiJKz1BBzHGH9jA2ykONzqB21HT9taBhFc 3F++eDMwiG4BFee66OSSZv20uXKhhpEmq2hFBIrDwBOTkDhdUpQTKcYqktuCBho81bU6 Ke6A== X-Forwarded-Encrypted: i=1; AJvYcCU7strb7qL3CFNPLlfp6QwSWHAPdgGDFkXxKC26cbQ5mATYvehrvQtwDK9EGVx0tsMayNkeZ9rsSyEphRmpDPq7K08= X-Gm-Message-State: AOJu0YyWfh8YRMEwz4l0Muu/OvXUnBZDR2+BEhb331VyjIGYvu4A7ix0 nV2t3ezgggvltnfUOeSR46mF8IzPiozASuqdxIHz3t8xOWL6xWiH X-Google-Smtp-Source: AGHT+IHR7uqhk7gVftB4fR9v+bC0QoXfZ4ovI4Z48TBsdfkzzx38SqwehsO7O7NBUo+/+reJ8o+fsg== X-Received: by 2002:a17:902:d38d:b0:1dc:8db3:16e9 with SMTP id e13-20020a170902d38d00b001dc8db316e9mr530965pld.45.1709167137495; Wed, 28 Feb 2024 16:38:57 -0800 (PST) Received: from localhost.localdomain ([2407:7000:8942:5500:5158:ed66:78b3:7fda]) by smtp.gmail.com with ESMTPSA id p3-20020a170902780300b001d9641003cfsm62647pll.142.2024.02.28.16.38.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 28 Feb 2024 16:38:57 -0800 (PST) From: Barry Song <21cnbao@gmail.com> To: akpm@linux-foundation.org, david@redhat.com, linux-mm@kvack.org, ryan.roberts@arm.com, chrisl@kernel.org Cc: 21cnbao@gmail.com, linux-kernel@vger.kernel.org, mhocko@suse.com, shy828301@gmail.com, steven.price@arm.com, surenb@google.com, wangkefeng.wang@huawei.com, willy@infradead.org, xiang@kernel.org, ying.huang@intel.com, yuzhao@google.com, kasong@tencent.com, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, hannes@cmpxchg.org, linux-arm-kernel@lists.infradead.org, Barry Song , Hugh Dickins , Minchan Kim , SeongJae Park Subject: [PATCH RFC v2 4/5] mm: swap: introduce swapcache_prepare_nr and swapcache_clear_nr for large folios swap-in Date: Thu, 29 Feb 2024 13:37:52 +1300 Message-Id: <20240229003753.134193-5-21cnbao@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240229003753.134193-1-21cnbao@gmail.com> References: <20240229003753.134193-1-21cnbao@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: A2E41180008 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 8cfbn6c7azmaoiceh7iusphrzojauthc X-HE-Tag: 1709167138-443586 X-HE-Meta: U2FsdGVkX1/HGlK4WlvYBWU+Tzv3/IzPnCEKY1LzHRP5+vwp1WZ07gspwTnqG82PoSqwHv4QOLTed4xkvmhgg+niVQgQtfXGd1aRrTRxiKL3zwvTMABhJHaR9b1Veq6/V5MhsTOFGWWJt2iUWMAI71/qHAPD1QQVmaqCvzsMaJvhpS+ySb15QMVYkaiS7kb49YtSaiXajYPnjGW8pRq1uWqgRZPYrWSlfmBmD3CnJ3TDAfDJMe9xujP1V1OuBOul2QYkV+ngqGtTQseq+MkZa5jCoFzs5cI7GyEYcbGC07DthwBaFeLjJRtOR+owiYnfCywXZy7BEpLBK59TVwVV1iUslJkubvvpbc0x9oGtvvFDxyRguw8Je8BgrTNsgzPboFo4qRSainT5/wmnyyie6PZUXqxvMH5TBc9QMamakQbJfWX6eGjVW15hiAvzgiMA5vBtucIzuqHsrpdnqMLz7EzMV6b9PnlUg74Rtsp+bmermb7rCIykVzQ52Hf1K7bWtM/Dd17BHg3BUONfUF3J0vAqRIE7UeCXr2+Nf61kOEPtA9412JAOg1DBcCmDaHGdso1vaSNh7DgX9xXyvAjeUqVez5/1mQZCf1rt7s9Sh2mfwQDcqh2ma5gZNvUNeFQDC2Rvy0hy6X7awD+4UyoCdzYnSQkVXkxY7z/DCFhhI3YVdz5Ia/fUGKCwrAABUWw6oqnkKWRcQq+RJFE7qHoUVx4N3WAz+xH/eC9uRB2u1Dz5o/VIC9I3uGNs9AeBTfraq9G/PTBydTyHipMrggTv9R/VrnPqqBtOcgDtU/82rf3T8Wg/FAuS5LHVt8BeQ8oErw2YXq+LXyIf36QAuiKJRpNpogOC7AGCwpSwZ3JVRQrYj3kciItCK6Vd86X6oGK96RI1LLoNqGnibDJsFAKd55AuZQ2uyOwKHwsnZb4MuXpfCYeFk1wSYF9xDqBVy3URWl/Rzq0BYlkHQ2iSpaJ kzhC8f6s B/qiNRb2woWCUNrA67e6+iSO+X8bijQtyua30155rYPD+2F1NbIJxQptc+DaOItFhs4UNFpLZiWaRIrerbZpLpv4lmZjg7GoGZlplsU1JvVCBtWN0adEc5vzIGBEIcV+JBtEb2k7TmlDe/6obenmryFjvRI/JVC3cRkpVAzwi67mcFcWka1REeofkg3OzN/ek6+F9K6YE9amA5odMEtfTNxQ8DTBhkCESNEEdaWUDWJ0YJs2+duucvXOSBIf9c5QBxpq6rUSnL07crhvkab3UnDoVROj/3p03nTEFV3eqK+b6Xw9xQ9ZgTM4Ivkf4N9CSZg7PYxFarin6yomhXz6jWG1JIVknC9PPfkoHrfj4gXzpLhYRnOYAJ/UQDl//7mD9BMvzg0q1FuKvlQn5pF4XdZKa8eYtZ8dh84X6okJSVMhJi1yYcRjUgoY4aiBCJdkeMUzQZrULykG1eMhPjBLVxkLiLQCdIzetFIgtv03QjfTxkjwL/S0xE8filmX+qjZgrC/Tl7f3G6n2lJUcqCufyx++DEsj4x27SnPRpHM3u9b5CZF5PApke53610pxQ+gjp2TLVCzgVrJyRq1wXCGrKTGuWKUul2xFftAppEmZg7QEi0SiC732TWDNacaX3ynSmnF3h8e6fsne4KhyDVhKgZFkc7fPTaWiRQ9T+S5acX8TV5yF9yv+LPYFZwoWzVxqdsUhkzKTMjZmI9vplTJso8bD6m9iLbr2tJrGWaAamEGBmUMf/KE6biBx1NM+ntYTy2jSYlGAZ9AbgD8jWCOOtmxrtkMYFGek2o0yLDImkycNW7OoiEkL+Ztpx+V3+4dihoLf7h3UUXOe8CekJKhii2mlnQV45xRfumUIlHCxd/mPsNnhGICvGWf86ZPLTW/Or2lr+b6+LDhUoNnnbbErXr2RjA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Barry Song Commit 13ddaf26be32 ("mm/swap: fix race when skipping swapcache") supports one entry only, to support large folio swap-in, we need to handle multiple swap entries. Cc: Kairui Song Cc: "Huang, Ying" Cc: Yu Zhao Cc: David Hildenbrand Cc: Chris Li Cc: Hugh Dickins Cc: Johannes Weiner Cc: Matthew Wilcox (Oracle) Cc: Michal Hocko Cc: Minchan Kim Cc: Yosry Ahmed Cc: Yu Zhao Cc: SeongJae Park Signed-off-by: Barry Song --- include/linux/swap.h | 1 + mm/swap.h | 1 + mm/swapfile.c | 117 ++++++++++++++++++++++++++----------------- 3 files changed, 72 insertions(+), 47 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index b3581c976e5f..2691c739d9a4 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -480,6 +480,7 @@ extern int add_swap_count_continuation(swp_entry_t, gfp_t); extern void swap_shmem_alloc(swp_entry_t); extern int swap_duplicate(swp_entry_t); extern int swapcache_prepare(swp_entry_t); +extern int swapcache_prepare_nr(swp_entry_t, int nr); extern void swap_free(swp_entry_t); extern void swap_nr_free(swp_entry_t entry, int nr_pages); extern void swapcache_free_entries(swp_entry_t *entries, int n); diff --git a/mm/swap.h b/mm/swap.h index fc2f6ade7f80..1cec991efcda 100644 --- a/mm/swap.h +++ b/mm/swap.h @@ -42,6 +42,7 @@ void delete_from_swap_cache(struct folio *folio); void clear_shadow_from_swap_cache(int type, unsigned long begin, unsigned long end); void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry); +void swapcache_clear_nr(struct swap_info_struct *si, swp_entry_t entry, int nr); struct folio *swap_cache_get_folio(swp_entry_t entry, struct vm_area_struct *vma, unsigned long addr); struct folio *filemap_get_incore_folio(struct address_space *mapping, diff --git a/mm/swapfile.c b/mm/swapfile.c index c0c058ee7b69..c8c8b6dbaeda 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -3308,7 +3308,7 @@ void si_swapinfo(struct sysinfo *val) } /* - * Verify that a swap entry is valid and increment its swap map count. + * Verify that nr swap entries are valid and increment their swap map count. * * Returns error code in following case. * - success -> 0 @@ -3318,66 +3318,73 @@ void si_swapinfo(struct sysinfo *val) * - swap-cache reference is requested but the entry is not used. -> ENOENT * - swap-mapped reference requested but needs continued swap count. -> ENOMEM */ -static int __swap_duplicate(swp_entry_t entry, unsigned char usage) +static int __swap_duplicate_nr(swp_entry_t entry, int nr, unsigned char usage) { struct swap_info_struct *p; struct swap_cluster_info *ci; unsigned long offset; - unsigned char count; - unsigned char has_cache; - int err; + unsigned char count[SWAPFILE_CLUSTER]; + unsigned char has_cache[SWAPFILE_CLUSTER]; + int err, i; p = swp_swap_info(entry); offset = swp_offset(entry); ci = lock_cluster_or_swap_info(p, offset); - count = p->swap_map[offset]; - - /* - * swapin_readahead() doesn't check if a swap entry is valid, so the - * swap entry could be SWAP_MAP_BAD. Check here with lock held. - */ - if (unlikely(swap_count(count) == SWAP_MAP_BAD)) { - err = -ENOENT; - goto unlock_out; - } + for (i = 0; i < nr; i++) { + count[i] = p->swap_map[offset + i]; - has_cache = count & SWAP_HAS_CACHE; - count &= ~SWAP_HAS_CACHE; - err = 0; - - if (usage == SWAP_HAS_CACHE) { - - /* set SWAP_HAS_CACHE if there is no cache and entry is used */ - if (!has_cache && count) - has_cache = SWAP_HAS_CACHE; - else if (has_cache) /* someone else added cache */ - err = -EEXIST; - else /* no users remaining */ + /* + * swapin_readahead() doesn't check if a swap entry is valid, so the + * swap entry could be SWAP_MAP_BAD. Check here with lock held. + */ + if (unlikely(swap_count(count[i]) == SWAP_MAP_BAD)) { err = -ENOENT; + goto unlock_out; + } - } else if (count || has_cache) { - - if ((count & ~COUNT_CONTINUED) < SWAP_MAP_MAX) - count += usage; - else if ((count & ~COUNT_CONTINUED) > SWAP_MAP_MAX) - err = -EINVAL; - else if (swap_count_continued(p, offset, count)) - count = COUNT_CONTINUED; - else - err = -ENOMEM; - } else - err = -ENOENT; /* unused swap entry */ - - if (!err) - WRITE_ONCE(p->swap_map[offset], count | has_cache); + has_cache[i] = count[i] & SWAP_HAS_CACHE; + count[i] &= ~SWAP_HAS_CACHE; + err = 0; + + if (usage == SWAP_HAS_CACHE) { + + /* set SWAP_HAS_CACHE if there is no cache and entry is used */ + if (!has_cache[i] && count[i]) + has_cache[i] = SWAP_HAS_CACHE; + else if (has_cache[i]) /* someone else added cache */ + err = -EEXIST; + else /* no users remaining */ + err = -ENOENT; + } else if (count[i] || has_cache[i]) { + + if ((count[i] & ~COUNT_CONTINUED) < SWAP_MAP_MAX) + count[i] += usage; + else if ((count[i] & ~COUNT_CONTINUED) > SWAP_MAP_MAX) + err = -EINVAL; + else if (swap_count_continued(p, offset + i, count[i])) + count[i] = COUNT_CONTINUED; + else + err = -ENOMEM; + } else + err = -ENOENT; /* unused swap entry */ + } + if (!err) { + for (i = 0; i < nr; i++) + WRITE_ONCE(p->swap_map[offset + i], count[i] | has_cache[i]); + } unlock_out: unlock_cluster_or_swap_info(p, ci); return err; } +static int __swap_duplicate(swp_entry_t entry, unsigned char usage) +{ + return __swap_duplicate_nr(entry, 1, usage); +} + /* * Help swapoff by noting that swap entry belongs to shmem/tmpfs * (in which case its reference count is never incremented). @@ -3416,17 +3423,33 @@ int swapcache_prepare(swp_entry_t entry) return __swap_duplicate(entry, SWAP_HAS_CACHE); } -void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry) +int swapcache_prepare_nr(swp_entry_t entry, int nr) +{ + return __swap_duplicate_nr(entry, nr, SWAP_HAS_CACHE); +} + +void swapcache_clear_nr(struct swap_info_struct *si, swp_entry_t entry, int nr) { struct swap_cluster_info *ci; unsigned long offset = swp_offset(entry); - unsigned char usage; + unsigned char usage[SWAPFILE_CLUSTER]; + int i; ci = lock_cluster_or_swap_info(si, offset); - usage = __swap_entry_free_locked(si, offset, SWAP_HAS_CACHE); + for (i = 0; i < nr; i++) + usage[i] = __swap_entry_free_locked(si, offset + i, SWAP_HAS_CACHE); unlock_cluster_or_swap_info(si, ci); - if (!usage) - free_swap_slot(entry); + for (i = 0; i < nr; i++) { + if (!usage[i]) { + free_swap_slot(entry); + entry.val++; + } + } +} + +void swapcache_clear(struct swap_info_struct *si, swp_entry_t entry) +{ + swapcache_clear_nr(si, entry, 1); } struct swap_info_struct *swp_swap_info(swp_entry_t entry)